Display apparatus, content recognizing method thereof, and non-transitory computer readable recording medium

ABSTRACT

A display apparatus, a content recognizing method thereof, and a non-transitory computer readable recording medium are provided. The display apparatus includes a display, a memory configured to store information regarding a fingerprint which is generated by extracting a characteristic of a content, and a content corresponding to the fingerprint, a communication device configured to communicate with a server, and at least one processor configured to extract a characteristic of a screen of a content currently reproduced on the display and generate a fingerprint, to search presence/absence of a fingerprint matching the generated fingerprint in the memory, and, based on a result of the searching, to determine whether to transmit a query comprising the generated fingerprint to the server to request information on the currently reproduced content.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser.No. 15/848,899, filed on Dec. 20, 2017, which was based on and claimedpriority under 35 U.S.C. § 119(a) of a Korean patent application number10-2016-0175741, filed on Dec. 21, 2016, in the Korean IntellectualProperty Office, and of a Korean patent application number10-2017-0133174, filed on Oct. 13, 2017, in the Korean IntellectualProperty Office, the disclosure of each of which is incorporated byreference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to a display apparatus, a contentrecognizing method thereof, and at least one non-transitory computerreadable recording medium. More particularly, the present disclosurerelates to a display apparatus which can efficiently recognize a contentviewed by a user, a content recognizing method thereof, and anon-transitory computer readable recording medium.

In addition, apparatuses and methods consistent with various embodimentsrelate to an artificial intelligence (AI) system which simulatesfunctions of the human brain, such as recognizing, determining, or thelike by utilizing a machine learning algorithm, such as deep learning,and application technology thereof.

BACKGROUND

In recent years, display apparatus, such as televisions (TVs) areincreasingly using set-top boxes rather than directly receivingbroadcast signals. In this case, a display apparatus cannot know whatcontent is currently viewed by a user.

If the display apparatus knows what content is currently viewed by theuser, smart services, such as targeting advertisements, contentrecommendation, related-information services can be provided. To achievethis, automatic content recognition (ACR) which is technology forrecognizing a currently displayed content at a display apparatus isdeveloped.

In a related-art method, a display apparatus periodically captures ascreen which is being currently viewed, extracts a characteristic forrecognizing the screen, and periodically requests a server to recognizethe current screen through a query.

However, the display apparatus has no choice but to frequently send aquery to the server in order to rapidly detect a change in the contentwhich is being viewed, and accordingly, ACR requires many resources andmuch cost.

With the development of computer technology, data traffic increases inthe form of an exponential function, and artificial intelligence (AI)becomes an important trend that leads future innovation. Since AIsimulates the way the human thinks, it can be applied to all industriesinfinitely.

The AI system refers to a computer system that implements highintelligence as human intelligence, and is a system that makes a machinelearn and determine by itself and become smarter unlike an existingrule-based smart system. The AI system can enhance a recognition rate asit is used and can exactly understand user's taste, and thus theexisting rule-based smart system is increasingly being replaced with adeep-learning based-AI system.

The AI technology includes machine learning (for example, deep learning)and element technology using machine learning.

Machine learning is algorithm technology for classifying//learningcharacteristics of input data by itself, and element technology istechnology for simulating functions of the human brain, such asrecognizing, determining, or the like by utilizing a machine learningalgorithm, such as deep learning, and may include technical fields, suchas linguistic understanding, visual understanding, inference/prediction,knowledge representation, operation control, or the like.

Various fields to which the AI technology is applied are as follows. Thelinguistic understanding is technology for recognizing humanlanguages/characters and applying/processing the same, and may includenatural language processing, machine translation, a dialog system,question and answer, voice recognition/synthesis. The visualunderstanding is technology for recognizing things in the same way ashumans do with eyes, and may include object recognition, objecttracking, image search, people recognition, scene understanding, spaceunderstanding, and image enhancement. The inference/prediction istechnology for inferring and predicting logically by determininginformation, and may include knowledge/probability-based inference,optimization prediction, preference-based planning, recommendation, orthe like. The knowledge representation is technology for automatinghuman experience information into knowledge data, and may includeknowledge establishment (data generation/classification), knowledgemanagement (data utilization), or the like. The operation control istechnology for controlling autonomous driving of vehicles and a motionof a robot, and may include motion control (navigation, collision,driving), manipulation control (behavior control), or the like.

The above information is presented as background information only toassist with an understanding of the present disclosure. No determinationhas been made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the present disclosure.

SUMMARY

Aspects of the present disclosure are to address at least the abovementioned problems and/or disadvantages and to provide at least theadvantages described below. Accordingly, an aspect of the presentdisclosure is to provide a display apparatus which can adjust a contentrecognition period using information of a content, a content recognizingmethod thereof, and a non-transitory computer readable recording medium.

In accordance with an aspect of the present disclosure, a displayapparatus is provided. The display apparatus includes a display, amemory configured to store information regarding a fingerprint which isgenerated by extracting a characteristic of a content, and a contentcorresponding to the fingerprint, a communication device configured tocommunicate with a server, and at least one processor configured toextract a characteristic of a screen of a content currently reproducedon the display and generate a fingerprint, to search presence/absence ofa fingerprint matching the generated fingerprint in the memory, and,based on a result of the searching, to determine whether to transmit aquery including the generated fingerprint to the server to requestinformation on the currently reproduced content.

The at least one processor may be configured to, in response to thefingerprint matching the generated fingerprint being searched in thememory, recognize the currently reproduced content based on informationon a content corresponding to the searched fingerprint, and the at leastone processor may be configured to, in response to the fingerprintmatching the generated fingerprint not being searched in the memory,control the communication device to transmit the query including thefingerprint to the server to request the information on the currentlyreproduced content.

The at least one processor may be configured to, in response to thefingerprint matching the generated fingerprint not being searched in thememory, control the communication device to receive the information onthe currently reproduced content and fingerprints of the currentlyreproduced content from the server in response to the query.

The at least one processor may be configured to determine a type of thecontent based on the information on the currently reproduced content,and to change a content recognition period according to the determinedtype of the content.

In addition, the at least one processor may be configured to recognize acontent in every first period in response to the content being anadvertisement content, and to recognize a content in every second periodwhich is longer than the first period in response to the content being abroadcast program content.

The at least one processor may be configured to determine a type of thecontent based on the information on the currently reproduced content,and to change a quantity of fingerprints of the currently reproducedcontent to be received according to the determined type of the content.

The at least one processor may be configured to calculate a probabilitythat the reproduced content is changed based on the information on thecurrently reproduced content and a viewing history, and to change acontent recognition period according to the calculated probability.

The at least one processor may be configured to predict a content to bereproduced next time based on a viewing history, and to requestinformation on the predicted content from the server.

The at least one processor may be configured to receive additionalinformation related to the currently reproduced content from the server,and to control the display to display the received additionalinformation with the currently reproduced content.

In accordance with another aspect of the present disclosure, a methodfor recognizing a content of a display apparatus is provided. The methodincludes extracting a characteristic of a screen of a currentlyreproduced content and generating a fingerprint, searching whether afingerprint matching the generated fingerprint is stored in the displayapparatus, and, based on a result of the searching, determining whetherto transmit a query including the generated fingerprint to an externalserver to request information on the currently reproduced content.

The determining whether to transmit the query to the external server mayinclude in response to the fingerprint matching the generatedfingerprint being searched in the display apparatus, recognizing thecurrently reproduced content based on information on a contentcorresponding to the searched fingerprint, and, in response to thefingerprint matching the generated fingerprint not being searched in thedisplay apparatus, transmitting the query including the fingerprint tothe server to request the information on the currently reproducedcontent.

In addition, the method may further include, in response to thefingerprint matching the generated fingerprint not being searched in thedisplay apparatus, receiving the information on the currently reproducedcontent and fingerprints of the currently reproduced content from theserver in response to the query.

The method may further include determining a type of the content basedon the information on the currently reproduced content, and changing acontent recognition period according to the determined type of thecontent.

The changing the content recognition period may include recognizing acontent in every first period in response to the content being anadvertisement content, and recognizing a content in every second periodwhich is longer than the first period in response to the content being abroadcast program content.

The method may further include determining a type of the content basedon the information on the currently reproduced content, and changing aquantity of fingerprints of the currently reproduced content to bereceived according to the determined type of the content.

The method may further include calculating a probability that thereproduced content is changed based on the information on the currentlyreproduced content and a viewing history, and changing a contentrecognition period according to the calculated probability.

The method may further include predicting a content to be reproducednext time based on a viewing history, and requesting information on thepredicted content from the server.

The method may further include receiving additional information relatedto the currently reproduced content from the server, and displaying thereceived additional information with the currently reproduced content.

In accordance with another aspect of the present disclosure, at leastone non-transitory computer readable recording medium is provided. Theat least one non-transitory computer readable recording medium includesa program for executing a method for recognizing a content of a displayapparatus, the method including extracting a characteristic of a screenof a currently reproduced content and generating a fingerprint,searching whether a fingerprint matching the generated fingerprint isstored in the display apparatus, and determining, based on a result ofthe searching, whether to transmit a query including the generatedfingerprint to an external server to request information on thecurrently reproduced content.

According to various embodiments described above, the display apparatusdynamically adjusts a ratio and a period of performance between serverautomatic content recognition (ACR) and local ACR, thereby reducing aload to the server and increasing accuracy in recognizing a content.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 illustrates a display system according to an embodiment of thepresent disclosure;

FIGS. 2A and 2B are schematic block diagrams illustrating aconfiguration of a display apparatus according to an embodiment of thepresent disclosure;

FIG. 3 is a block diagram of a processor according to an embodiment ofthe present disclosure;

FIG. 4A is a block diagram of a data learning unit according to anembodiment of the present disclosure;

FIG. 4B is a block diagram of a data recognition unit according to anembodiment of the present disclosure;

FIG. 5 is a block diagram illustrating a configuration of a displayapparatus according to an embodiment of the present disclosure;

FIG. 6 is a view illustrating hybrid automatic content recognition (ACR)according to an embodiment of the present disclosure;

FIGS. 7A and 7B are views illustrating fingerprint information havingdifferent granularities according to an embodiment of the presentdisclosure;

FIG. 8 is a view illustrating viewing history information according toan embodiment of the present disclosure;

FIG. 9 is a view illustrating display of additional information with acontent according to an embodiment of the present disclosure;

FIGS. 10, 11, 12A, 12B, 13A, 13B, 14A, 14B, 15A, and 15B are flowchartsillustrating a content recognizing method of a display apparatusaccording to various embodiments of the present disclosure;

FIG. 16 is a view illustrating data being learned and recognized by adisplay apparatus and a server interlocked with each other according toan embodiment of the present disclosure;

FIG. 17 is a flowchart illustrating a content recognizing method of adisplay system according to an embodiment of the present disclosure;

FIG. 18 is a flowchart illustrating a content recognizing method of adisplay system according to an embodiment of the present disclosure;

FIG. 19 is a view illustrating a situation in which a display apparatuschanges a content recognition period according to a probability that acontent is changed by interlocking with a server according to anembodiment of the present disclosure;

FIG. 20 is a view illustrating a method by which a display apparatuspredicts a content to be reproduced next time, and receives informationon a predicted content in advance by interlocking with a serveraccording to an embodiment of the present disclosure; and

FIG. 21 is a view illustrating a method by which a display apparatuspredicts a content to be reproduced next time, and receives informationon a predicted content in advance by interlocking with a plurality ofservers according to an embodiment of the present disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the present disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of the presentdisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of the presentdisclosure is provided for illustration purpose only and not for thepurpose of limiting the present disclosure as defined by the appendedclaims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

By the term “substantially” it is meant that the recited characteristic,parameter, or value need not be achieved exactly, but that deviations orvariations, including for example, tolerances, measurement error,measurement accuracy limitations and other factors known to those ofskill in the art, may occur in amounts that do not preclude the effectthe characteristic was intended to provide.

The terms, such as “first” and “second” used in various embodiments maybe used to explain various elements, but do not limit the correspondingelements. These terms may be used for the purpose of distinguishing oneelement from another element. For example, a first element may be nameda second element without departing from the scope of right of thepresent disclosure, and similarly, a second element may be named a firstelement. The term “and/or” includes a combination of a plurality ofrelated items or any one of the plurality of related items.

The terms used in various embodiments of the present disclosure are justfor the purpose of describing particular embodiments and are notintended to restrict and/or limit the present disclosure. As usedherein, the singular forms are intended to include the plural forms aswell, unless the context clearly indicates otherwise. The term “include”or “have” indicates the presence of features, numbers, operations,elements, and components described in the specification, or acombination thereof, and do not preclude the presence or addition of oneor more other features, numbers, operation, elements, or components, ora combination thereof.

In addition, a “module” or “unit” used in embodiments performs one ormore functions or operations, and may be implemented by using hardwareor software or a combination of hardware and software. In addition, aplurality of “modules” or a plurality of “units” may be integrated intoone or more modules, except for a “module” or “unit” which needs to beimplemented by specific hardware, and may be implemented as one or moreprocessors.

Hereinafter, the present disclosure will be described below withreference to the accompanying drawings.

FIG. 1 illustrates a display system according to an embodiment of thepresent disclosure.

Referring to FIG. 1, a display system 1000 includes a display apparatus100 and a server 200.

The display apparatus 100 may be a smart television (TV), but this ismerely an example, and the display apparatus 100 may be implemented byusing various types of apparatuses, such as a projection TV, a monitor,a kiosk, a notebook personal computer (PC), a tablet, a smartphone, apersonal digital assistant (PDA), an electronic picture frame, a tabledisplay, or the like.

The display apparatus 100 may extract a characteristic from a screen ofa currently reproduced content, and may generate a fingerprint. Inaddition, the display apparatus 100 may perform local automatic contentrecognition (ACR) by searching the generated fingerprint in afingerprint database stored in the display apparatus 100, andrecognizing the currently reproduced content, and may perform server ACRby transmitting a query including the generated fingerprint to theserver 200 and recognizing the content. More particularly, the displayapparatus 100 may appropriately adjust a ratio of performance betweenthe local ACR and the server ACR by adjusting a content recognitionperiod, a quantity of fingerprints to be received from the server 200,or the like using recognized content information, a viewing history, orthe like.

The server 200 may be implemented by using an apparatus that cantransmit information including recognition (or identification (ID))information for distinguishing a specific image from other images to thedisplay apparatus 100. For example, the server 200 may transmit afingerprint to the display apparatus 100. The fingerprint is a kind ofidentification information that can distinguish an image from otherimages.

Specifically, the fingerprint is characteristic data that is extractedfrom video and audio signals included in a frame. Unlike metadata basedon a text, the fingerprint may reflect unique characteristics of asignal. For example, when an audio signal is included in the frame, thefingerprint may be data representing characteristics of the audiosignal, such as a frequency, an amplitude, or the like. When a video (orstill image) signal is included in the frame, the fingerprint may bedata representing characteristics, such as a motion vector, color, orthe like.

The fingerprint may be extracted by various algorithms. For example, thedisplay apparatus 100 or the server 200 may divide an audio signalaccording to regular time intervals, and may calculate a size of asignal of frequencies included in each time interval. In addition, thedisplay apparatus 100 or the server 200 may calculate a frequency slopeby obtaining a difference in size between signals of adjacent frequencysections. A fingerprint on the audio signal may be generated by setting1 when the calculated frequency slope is a positive value and setting 0when the calculated frequency slope is a negative value.

The server 200 may store a fingerprint on a specific image. The server200 may store one or more fingerprints on an already registered image,and, when two or more fingerprints are stored regarding a specificimage, the fingerprints may be managed as a fingerprint list for thespecific image.

The term “fingerprint” used in the present disclosure may refer to onefingerprint on a specific image, or according to circumstances, mayrefer to a fingerprint list which is formed of a plurality offingerprints on a specific image.

The term “frame” used in the present disclosure refers to a series ofdata having information on an audio or an image. The frame may be dataon an audio or an image during a predetermined time. In the case of adigital image content, the frame may be formed of 30-60 image data persecond, and these 30-60 image data may be referred to as a frame. Forexample, when a current frame and a next frame of an image content areused together, the frame may refer to respective image screens includedin the content and continuously displayed.

Although FIG. 1 depicts that the display system 1000 includes onedisplay apparatus 100 and one server 200, a plurality of displayapparatuses 100 may be connected with one server 200 or a plurality ofservers 200 may be connected with one display apparatus 100. Othercombinations are also possible.

FIG. 2A is a block diagram illustrating a configuration of a displayapparatus according to an embodiment of the present disclosure.

Referring to FIG. 2A, the display apparatus 100 may include a display110, a memory 120, a communication unit 130, and a processor 140.

The display 110 may display various image contents, information, a userinterface (UI), or the like which are provided by the display apparatus100. For example, the display apparatus 100 may display an imagecontent, a broadcast program image, a user interface window which areprovided by a set-top box (not shown).

The memory 120 may store various modules, software, and data for drivingthe display apparatus 100. For example, the memory 120 may store aplurality of fingerprints, information on contents corresponding to thefingerprints, viewing history information, or the like. The fingerprintsstored in the memory 120 may be those which are generated by the displayapparatus 100 itself, or may be those which are received from the server200. The memory 120 may attach index information for local ACR to thefingerprints and store the fingerprints.

In addition, when a content stored in the memory 120 is reproduced bythe display apparatus 100, a fingerprint may be paired with thecorresponding content and may be stored in the memory 120. For example,the fingerprint may be added to each frame of the content and may bestored in the form of a new file combining the content and thefingerprint. In another example, the fingerprint may further includeinformation mapped onto a corresponding frame of the content.

The communication unit 130 may communicate with external devices, suchas the server 200 using a wire/wireless communication method. Forexample, the communication unit 130 may exchange, with the server 200,data, such as a fingerprint, content information, viewing historyinformation, additional information related to a content, and a controlsignal, such as a recognition period change control signal.

The processor 140 may recognize what content is currently reproduced,and may control to perform ACR with appropriate precision based on theresult of recognizing. For example, the processor 140 may adjust acontent recognition period based on content information, and maydetermine a content to be received from the server 200 in advance andstored and a quantity of fingerprints on the content.

The processor 140 may extract a characteristic of a screen of thecurrently reproduced content, and may generate a fingerprint. Inaddition, the processor 140 may search whether there is a fingerprintmatching the generated fingerprint from among the plurality offingerprints stored in the memory 120. In addition, the processor 140may transmit a query to the server 200 according to the result ofsearching, and may determine whether to try to perform server ACR. Forexample, the processor 140 may try to perform local ACR, first, in orderto reduce a load on the server 200.

In response to a fingerprint matching the generated fingerprint beingsearched, the processor 140 may recognize the currently reproducedcontent based on information on the content corresponding to thesearched fingerprint. For example, the information on the contentcorresponding to the fingerprint may include a position of a currentframe in the total frames, a reproducing time, or the like, which areinformation on the current frame. In addition, the information on thecontent corresponding to the fingerprint may include at least one of acontent name, a content ID, a content provider, content seriesinformation, a genre, information on whether the content is a real-timebroadcast, and information on whether the content is a paid content.

On the other hand, in response to the fingerprint matching the generatedfingerprint not being searched, the processor 140 may control thecommunication unit 130 to transmit a query for requesting information onthe currently reproduced content to the server 200. For example, thequery may include the generated fingerprint, a viewing history,information on the display apparatus 100, or the like.

In addition, the processor 140 may control the communication unit 130 toreceive the information on the currently reproduced content and afingerprint of the currently reproduced content from the server 200 inresponse to the query. Herein, the fingerprint of the currentlyreproduced content may be a fingerprint regarding frames which arepositioned after a current frame in time in the whole content. Since theprocessor 140 knows time indicated by the position of the current framein the whole content based on the fingerprint included in the query, theprocessor 140 may receive, from the server, a fingerprint on frameswhich are expected to be reproduced after the current frame.

As described above, the processor 140 may recognize the content bycombining the local ACR and the server ACR appropriately. By doing so,the processor 140 may recognize the content currently reproduced on thedisplay 110 while minimizing a load on the server 200.

To appropriately combine the local ACR and the server ACR, the processor140 may determine a fingerprint to be received from the server 200 inadvance for the local ACR based on at least one of the result of contentrecognition and the viewing history, and may determine whether to changethe content recognition period. For example, the processor 140 maydetermine what content is considered to receive the fingerprint, and howmany fingerprints will be received at a time.

According to an embodiment of the present disclosure, the processor 140may determine a type of the content based on the information on thecurrently reproduced content. In addition, the processor 140 may changethe content recognition period according to the type of the content. Thetype of the content may be classified according to a criterion, such asdetails of the content, a genre, information on whether the content is areal-time broadcast, an importance, or the like.

For example, in response to the currently reproduced content beingrecognized as an advertisement content, the processor 140 may adjust thecontent recognition period to be short (for example, adjust to recognizethe screen of the currently displayed content in every frame). Inresponse to the currently reproduced content being recognized as a moviecontent or a broadcast program content, the processor 140 may adjust thecontent recognition period to be long (for example, adjust to recognizethe screen of the currently displayed content once every 30 seconds).

The recognition period for each type may be a predetermined period. Therespective periods may be personalized and set according to theabove-described various criteria and the viewing history.

In the case of an advertisement content, the processor 140 needs tofrequently recognize the content since an advertisement is normallychanged to another advertisement within a short time. To the contrary,in the case of a movie content, the processor 140 does not need tofrequently recognize the content since it is just determined whether themovie is being continuously viewed.

In the above example, the type of the content is classified according tothe genre of the content. As in the above example, in the case of theadvertisement content, the content may be recognized in every frame, butmay be infrequently recognized according to other criteria, such as animportance or a viewing history.

The number of frames to be received from the server 200 in advance andstored may vary according to the recognized type of the content. Sincethere are fingerprints corresponding to the respective frames, thequantity of fingerprints to be received in advance and stored may alsovary. For example, in the case of video on demand (VOD) or digital videorecorder (DVR), the server 200 may have all pieces of image information,but in the case of a live broadcast, the server 200 may receive imageinformation of a few seconds before the display apparatus 100 does.Let's take an example of a 60 Hz image displaying 60 frames per second.In the case of one hour of VOD, the server 200 may own fingerprintscorresponding to about 200,000 frames, but in the case of a livebroadcast, the server 200 may only own fingerprints corresponding toabout hundreds of frames.

Accordingly, the processor 140 may determine a quantity of fingerprintsto be requested according to the recognized content. In addition, theprocessor 140 may change the content recognition period based on aquantity of fingerprints received at a time.

According to an embodiment of the present disclosure, the processor 140may change the content recognition period based on the information onthe recognized content and the viewing history. The viewing history mayinclude a content that the user has viewed, a viewing time, anadditional application which has been executed at the time of viewing.

For example, the processor 140 may determine whether the currentlyreproduced content will continuously be reproduced or another contentwill be reproduced by comparing the currently reproduced content and theviewing history. In addition, the processor 140 may request informationon a fingerprint corresponding to the content which is expected to bereproduced next time from the server 200.

According to an embodiment of the present disclosure, the processor 140may receive additional information related to the content, in additionto the fingerprint of the currently reproduced content and thefingerprint corresponding to the content which is expected to bereproduced next time. For example, the additional information mayinclude a content name, a content reproducing time, a content provider,PPL product information appearing in the content, an advertisementrelated to the PPL product information, and an additional executableapplication, or the like.

In addition, the processor 140 may control the display 110 to displaythe received additional information with the currently reproducedcontent.

In the above-described examples, the display apparatus 100 requests theinformation of the content, such as the fingerprint from the server 200.However, the server 200 may transmit the information necessary for thedisplay apparatus 100, such as the fingerprint, to the display apparatus100 without receiving a request.

According to various embodiments of the present disclosure, the displayapparatus 100 may estimate the type of the content using a datarecognition model. The data recognition model may be, for example, a setof algorithms for estimating a type of a content using information ofthe content and/or a fingerprint generated from the content, using aresult of statistical machine learning.

In addition, the display apparatus 100 may calculate a probability thatthe content is changed using the data recognition model. The datarecognition model may be, for example, a set of algorithms forestimating a probability that the reproduced content is changed usingthe information of the content (for example, a content reproducing time,a content reproducing channel, a type of a content, or the like).

The data recognition model may be implemented by using software or anengine for executing the set of algorithms. The data recognition modelimplemented by using the software or engine may be executed by theprocessor in the display apparatus 100 or a processor of a server (forexample, the server 200 in FIG. 1).

According to an embodiment of the present disclosure, the server 200 mayinclude configurations of a normal server device. For example, theserver 200 may include a memory 210, a communication unit 220, abroadcast signal receiver 230, and a processor 240.

The server 200 may capture video/audio information of a plurality ofcontents. For example, the server 200 may collect an image on a framebasis. For example, the server 200 may divide various contents into dataof a frame unit in advance, and may collect the data. In addition, theserver 200 may generate a fingerprint by analyzing the collected frames.In another example, the server 200 may receive a broadcast signal from abroadcasting station and may capture video/audio information from thereceived signal. The server 200 may receive the broadcast signal beforethe display apparatus 100 does.

For example, the fingerprint generated by the server 200 may beinformation for distinguishing between a screen and an audio at aspecific time. Alternatively, the fingerprint generated by the server200 may include information on a scene change pattern, and may beinformation indicating what content is being continuously viewed. Theserver 200 may establish a database in which the generated fingerprintand the information on the content corresponding to the fingerprint areindexed to be easily searched. For example, the information on thecontent corresponding to the fingerprint may include a position of acurrent frame in the whole content, a reproducing time, or the like. Inaddition, the information on the content corresponding to thefingerprint may include at least one of a content name, a content ID, acontent provider, content series information, a genre, information onwhether the content is a real-time broadcast, information on whether thecontent is a paid content.

In response to a query being received from the display apparatus 100,the server 200 may extract at least one fingerprint from the query. Inaddition, the server 200 may receive information on the displayapparatus 100 which has transmitted the query.

The server 200 may match the extracted fingerprint with informationstored in the database, and may determine what content is beingcurrently viewed by the display apparatus 100. The server 200 maytransmit a response on the determined content information to the displayapparatus 100.

In addition, the server 200 may manage the viewing history of each ofthe display apparatuses 100 using the received information on thedisplay apparatus 100 and the determined content information. By doingso, the server 200 may provide a service which is personalized for eachof the display apparatuses 100.

The server 200 may predict a content to be displayed on the displayapparatus 100 next time, using the information of the content currentlydisplayed on the display apparatus 100 and the viewing historyinformation. In addition, the server 200 may transmit a fingerprintwhich is extracted from the predicted content to the display apparatus100. For example, the extracted fingerprint may be a fingerprintcorresponding to a frame which is positioned after the current frame intime in the whole content displayed on the display apparatus 100. Inanother example, the extracted fingerprint may be a fingerprint on acontent of another broadcast channel which is predicted based on theviewing history information.

In addition, the server 200 may determine a content recognition periodby analyzing a content image or using electronic program guide (EPG)information. According to the determined content recognition period, theserver 200 may determine the number of fingerprints necessary forperforming local ACR at the display apparatus 100. The display apparatus100 may generate a fingerprint by analyzing a currently displayed framein every content recognition period. In addition, the display apparatus100 may search the generated fingerprint in the fingerprint data basewhich is received from the server 200 and stored. Accordingly, theserver 200 may transmit only the fingerprint corresponding to the framefor the display apparatus 100 to recognize the content. Since only thenecessary number of fingerprints are transmitted, the server 200 mayminimize a communication load even when performing server ACR.

FIG. 2B illustrates a configuration of a display apparatus according toan embodiment of the present disclosure.

Referring to FIG. 2B, the display apparatus 100 may include a firstprocessor 140-1, a second processor 140-2, a display 110, a memory 120,and a communication unit 130. However, all of the elements shown in thedrawing are not essential elements.

The first processor 140-1 may control execution of at least oneapplication installed in the display apparatus 100. For example, thefirst processor 140-1 may generate a fingerprint by capturing an imagedisplayed on the display 110, and may perform ACR. The first processor140-1 may be implemented in the form of a system on chip (SoC)integrating functions of a central processing unit (CPU), a graphicprocessing unit (GPU), a communication chip, and a sensor.Alternatively, the first processor 140-1 may be an application processor(AP).

The second processor 140-2 may estimate a type of a content using a datarecognition model. The data recognition model may be, for example, a setof algorithms for estimating the type of the content using theinformation of the content and/or the fingerprint generated from thecontent, using a result of statistical machine learning.

In addition, the second processor 140-2 may calculate a probability thatthe content is changed using the data recognition model. The datarecognition model may be, for example, a set of algorithms forestimating a probability that the reproduced content is changed usingthe information of the content (for example, a content reproducing time,a content reproducing channel, a type of a content, or the like), and aviewing history.

In addition, the second processor 140-2 may estimate a content to bereproduced next time after the reproduced content using the datarecognition model. The data recognition model may be, for example, a setof algorithms for estimating a probability that the reproduced contentis changed using the information of the content (for example, a contentreproducing time, a content reproducing channel, a type of a content, orthe like), and the viewing history.

The second processor 140-2 may be manufactured in the form of adedicated hardware chip for AI which performs the functions ofestimating the type of the content and estimating the probability thatthe content is changed using the data recognition model.

According to an embodiment of the present disclosure, the firstprocessor 140-1 and the second processor 140-2 may be interlocked witheach other to perform a series of processes as the processor 140 does togenerate a fingerprint from the content and recognize the content usingACR as described above with reference to FIG. 2A.

The display 110, the memory 120, and the communication unit 130correspond to the display 110, the memory 120, and the communicationunit 130 in FIG. 2A, respectively, and thus a redundant explanationthereof is omitted.

FIG. 3 is a block diagram of a processor according to an embodiment ofthe present disclosure.

Referring to FIG. 3, the processor 140 according to an embodiment mayinclude a data learning unit 141 and a data recognition unit 142.

The data learning unit 141 may learn in order for the data recognitionmodel to have a criterion for analyzing characteristics of predeterminedvideo/audio data. The processor 140 may generate a fingerprint byanalyzing characteristics of each of the captured frames (for example, achange in a frequency of audio data, a change in color of each frame ofvideo data, or a change in a motion vector) according to the learnedcriterion.

The data learning unit 141 may determine what learning data will be usedto determine the characteristics of the screen (or frame) of thecaptured content. In addition, the data learning unit 141 may learn thecriterion for extracting the characteristics of the captured contentusing the determined learning data.

According to various embodiments of the present disclosure, the datalearning unit 141 may learn in order for the data recognition model tohave a criterion for estimating a type of the video/audio data based onlearning data related to information on the predetermined video/audiodata and the type of the video/audio data.

The information on the video/audio data may include, for example, aposition of the current frame in the whole video/audio and a reproducingtime, which are information on the current frame. In addition, theinformation on the video/audio may include at least one of a video/audioname, a video/audio ID, a video/audio provider, video/audio seriesinformation, a genre, information on whether the video/audio is areal-time broadcast, information on whether the video/audio is a paidcontent.

The type of the video data may include, for example, drama,advertisement, movie, news, or the like. The type of the audio data mayinclude, for example, music, news, advertisement, or the like. However,the type of the audio/video data is not limited thereto.

According to various embodiments of the present disclosure, the datalearning unit 141 may learn in order for the data recognition model tohave a criterion for estimating a probability that video/audio data ischanged to another video/audio data during reproduction, or a criterionfor estimating video/audio data to be reproduced next time after thereproduction is completed, based on learning data related to theinformation on the predetermined video/audio data, the type of thevideo/audio data, and a viewing history of the video/audio data (forexample, a history of having changed to another video/audio data toview).

The data recognition unit 142 may recognize a situation based onpredetermined recognition data using the learned data recognition model.The data recognition unit 142 may obtain the predetermined recognitiondata according to a predetermined criterion obtained according tolearning, and may use the data recognition model using the obtainedrecognition data as an input value.

For example, using a learned characteristic extraction model, the datarecognition unit 142 may extract characteristics on respective framesincluded in the recognition data, such as a captured content, and maygenerate a fingerprint. In addition, the data recognition unit 142 mayupdate the data recognition model using output data which is obtained asa result of applying the data recognition model as an input value again.

According to various embodiments of the present disclosure, the datarecognition unit 142 may obtain a result of determining the type of thevideo/audio data by applying, to the data recognition model, therecognition data related to the information on the predeterminedvideo/audio data as an input value.

According to various embodiments of the present disclosure, the datarecognition unit 142 may obtain a result of estimating a probabilitythat video/audio data is changed to another video/audio data while thevideo/audio data is reproduced, or a result of estimating video/audiodata to be reproduced next time after the reproduction is completed, byapplying, to the data recognition model, the recognition data related tothe information on the predetermined video/audio data and the type ofthe video/audio data as an input value.

At least one of the data learning unit 141 and the data recognition unit142 may be manufactured in the form of one hardware chip or a pluralityof hardware chips and may be mounted in the display apparatus 100. Forexample, at least one of the data learning unit 141 and the datarecognition unit 142 may be manufactured in the form of a dedicatedhardware chip for AI, or may be manufactured as a part of an existinggeneric-purpose processor (for example, a CPU or an AP) or a part of agraphic dedicated processor (for example, a GPU, an ISP), and may bemounted in the above-described various display apparatuses 100.

In this case, the dedicated hardware chip for AI may be a dedicatedprocessor which is specialized in calculation of a probability, and mayhave higher parallel processing performance than that of the existinggeneral processor and thus can rapidly process an operation task of thefield of AI, such as machine learning. When the data learning unit 141and the data recognition unit 142 are implemented by using a softwaremodule (or a program module including instructions), the software modulemay be stored in a non-transitory computer readable recording medium. Inthis case, the software module may be provided by an operating system(OS) or may be provided by a predetermined application. Alternatively, apart of the software module may be provided by the OS and the other partmay be provided by the predetermined application.

Although FIG. 3 depicts that the data learning unit 141 and the datarecognition unit 142 are all mounted in the display apparatus 100, theymay be mounted in separate devices. For example, one of the datalearning unit 141 and the data recognition unit 12 may be included inthe display apparatus 100, and the other one may be included in theserver 200. In addition, the data learning unit 141 and the datarecognition unit 142 may be connected with each other in a wire orwireless method, and model information established by the data learningunit 141 may be provided to the data recognition unit 142, and datainputted to the data recognition unit 142 may be provided to the datalearning unit 141 as additional learning data.

FIG. 4A is a block diagram of a data learning unit according to anembodiment of the present disclosure.

Referring to FIG. 4A, the data learning unit 141 according to anembodiment may include a data obtaining unit 141-1 and a model learningunit 141-4. In addition, the data learning unit 141 may furtherselectively include at least one of a pre-processing unit 141-2, alearning data selection unit 141-3, and a model evaluation unit 141-5.

The data obtaining unit 141-1 may obtain learning data necessary fordetermining a situation. For example, the data obtaining unit 141-1 mayobtain an image frame by capturing a screen reproduced on the display110. In addition, the data obtaining unit 141-1 may receive image datafrom an external device, such as a set-top box. The image data may beformed of a plurality of image frames. In addition, the data obtainingunit 141-1 may receive learning image data from the server 200 or anetwork, such as the Internet.

The model learning unit 141-4 may learn in order for the datarecognition model to have a criterion for determining a situation basedon learning data. In addition, the model learning unit 141-4 may learnin order for the data recognition model to have a criterion forselecting what learning data will be used to determine the situation.

For example, the model learning unit 141-4 may learn physicalcharacteristics for distinguishing images by comparing the plurality ofimage frames. The model learning unit 141-4 may learn a criterion fordistinguishing image frames by extracting a ratio between a foregroundand a background in the image frame, a size, a location, and arrangementof an object, and characteristic points.

In addition, the model learning unit 141-4 may learn a criterion foridentifying a genre of a content including image frames. For example,the model learning unit 141-4 may learn a criterion for identifyingframes having a text box on an upper end or a lower of the left of theimage frame as one genre. This is because images of a news content havea text box on the upper end or lower end of the left side to show thenews content.

According to various embodiments of the present disclosure, the modellearning unit 141 may learn in order for the data recognition model tohave a criterion for estimating a type of video/audio data based onlearning data related to information of predetermined video/audio dataand a type of video/audio data.

The information of the video/audio data may include, for example, aposition of a current frame in the whole video/audio and a reproducingtime, which are information on the current frame. In addition, theinformation on the video/audio may include at least one of a video/audioname, a video/audio ID, a video/audio provider, video/audio seriesinformation, a genre, information on whether the video/audio is areal-time broadcast, information on whether the video/audio is a paidcontent.

The type of the video data may include, for example, drama,advertisement, movie, news, or the like. The type of the audio data mayinclude, for example, music, news, advertisement, or the like. However,the type of the audio/video data is not limited thereto.

According to various embodiments of the present disclosure, the modellearning unit 141 may learn in order for the data recognition model tohave a criterion for estimating a probability that video/audio data ischanged to another video/audio data during reproduction based onlearning data related to the information of the predeterminedvideo/audio data, the type of the video/audio data, and a viewinghistory of the video/audio data (for example, a history of havingchanged to another audio/video data or a history of having selectedanother audio/video data after viewing of the audio/video data wasfinished), or a criterion for estimating video/audio data to bereproduced next time after the reproduction is completed.

The data recognition model may be an already established model. Forexample, the data recognition model may be a model which receives basiclearning data (for example, a sample image) and is already established.

The data learning unit 141 may further include the pre-processing unit141-2, the learning data selection unit 141-3, and the model evaluationunit 141-5 in order to improve the result of recognizing of the datarecognition model or in order to save resources or time necessary forgenerating the data recognition model.

The pre-processing unit 141-2 may pre-process the obtained learning datasuch that the obtained learning data can be used for learning fordetermining a situation. The pre-processing unit 141-2 may processobtained data in a predetermined format such that the model learningunit 141-4 can use the obtained data for learning for determining asituation.

For example, the pre-processing unit 141-2 may generate image frames ofthe same format by performing decoding, scaling, noise filtering,resolution conversion, or the like with respect to inputted image data.In addition, the pre-processing unit 141-2 may crop only a specificregion included in each of the inputted plurality of image frames. Inresponse to only the specific region being cropped, the displayapparatus 100 may distinguish one of the frames from the others byconsuming fewer resources.

In another example, the pre-processing unit 141-2 may extract a textregion included in the inputted image frames. In addition, thepre-processing unit 141-2 may generate text data by performing opticalcharacter recognition (OCR) with respect to the extracted text region.The text data pre-processed as described above may be used todistinguish the image frames.

The learning data selection unit 141-3 may select data which isnecessary for learning from among the pre-processed data. The selecteddata may be provided to the model learning unit 141-4. The learning dataselection unit 141-3 may select data which is necessary for learningfrom among the pre-processed data according to a predetermined criterionfor determining a situation. In addition, the learning data selectionunit 141-3 may select data according to a predetermined criterion whichis determined by learning by the model learning data selection unit141-3, which will be described below.

For example, at the initial time of learning, the learning dataselection unit 141-3 may remove image frames having high similarity fromthe pre-processed image frames. For example, the learning data selectionunit 141-3 may select data having low similarity for initial learning,such that a criterion easy to learn can be learned.

In addition, the learning data selection unit 141-3 may selectpre-processed image frames which commonly satisfy one of criteriadetermined by leaning By doing so, the model learning unit 141-4 maylearn a different criterion from the already learned criterion.

The model evaluation unit 141-5 may input evaluation data into the datarecognition model, and, in response to a result of recognition outputtedfrom the evaluation data not satisfying a predetermined criterion, mayhave the model learning unit 141-4 learn again. In this case, theevaluation data may be predetermined data for evaluating the datarecognition model.

At an initial recognition model configuration operation, the evaluationdata may be image frames representing different content genres.Thereafter, the evaluation data may be substituted with a set of imageframes having higher similarity. By doing so, the model evaluation unit141-5 may verify performance of the data recognition model in phases.

For example, in response to the number or ratio of evaluation dataresulting inexact recognition results, from among the recognitionresults of the learned data recognition model regarding the evaluationdata, exceeding a predetermined threshold, the model evaluation unit141-5 may evaluate that the predetermined criterion is not satisfied.For example, the predetermined criterion may be defined as a ratio of2%. In this case, in response to the learned data recognition modeloutputting wrong recognition results with respect to 20 or moreevaluation data from among 1000 total evaluation data, the modelevaluation unit 141-5 may evaluate that the learned data recognitionmodel is not appropriate.

In response to there being a plurality of learned data recognitionmodels, the model evaluation unit 141-5 may evaluate whether each of thelearned data recognition models satisfies the predetermined criterion,and may determine a model satisfying the predetermined criterion as afinal data recognition model. In this case, in response to a pluralityof models satisfying the predetermined criterion, the model evaluationunit 141-5 may determine any predetermined model or a predeterminednumber of models as final data recognition models in order from thehighest evaluation score.

The data recognition model may be established based on an applicationfield of the recognition model, a purpose of learning, or computerperformance of a device. The data recognition model may be based on, forexample, a neural network. For example, models, such as a deep neuralnetwork (DNN), a recurrent neural network (RNN), a bidirectionalrecurrent deep neural network (BRDNN) may be used as the datarecognition model, but the data recognition model is not limitedthereto.

According to various embodiments of the present disclosure, in responseto there being a plurality of data recognition models alreadyestablished, the model learning unit 141-4 may determine a datarecognition model having high relevance to inputted learning data andbasic learning data as the data recognition model for learning. In thiscase, the basic learning data may be already classified according to atype of data, and the data recognition model may be already establishedaccording to a type of data. For example, the basic learning data may bealready classified according to various criteria, such as a region wherelearning data is generated, a time at which learning data is generated,a size of learning data, a genre of learning data, a generator oflearning data, a type of an object in learning data, or the like.

In addition, the model learning unit 141-4 may have the data recognitionmodel learn by using a learning algorithm including an error backpropagation or gradient descent method, for example.

For example, the model learning unit 141-4 may have the data recognitionmodel learn through supervised learning which considers learning datafor learning to have a determination criterion as an input value. Inanother example, the model learning unit 141-4 may learn a type of datanecessary for determining a situation by itself without separatesupervision, thereby having the data recognition model learn throughunsupervised learning which finds a criterion for determining asituation. In another example, the model learning unit 141-4 may havethe data recognition model learn through reinforcement learning whichuses feedback regarding whether a result of determining a situationaccording to learning is correct.

In addition, in response to the data recognition model being learned,the model learning unit 141-4 may store the learned data recognitionmodel. In this case, the model learning unit 141-4 may store the learneddata recognition model in the memory 120 of the display apparatus 100.Alternatively, the model learning unit 141-4 may store the learned datarecognition model in the memory of the server 200 connected with anelectronic device in a wire or wireless network.

In this case, the memory 120 in which the learned data recognition modelis stored may also store instructions or data related to at least oneother element of the display apparatus 100. In addition, the memory 120may store software and/or programs. For example, the programs mayinclude a kernel, middleware, an application programming interface(API), and/or an application program (or an application).

At least one of the data obtaining unit 141-1, the pre-processing unit141-2, the learning data selection unit 141-3, the model learning unit141-4, and the model evaluation unit 141-5 included in the data learningunit 141 may be manufactured in the form of at least one hardware chip,and may be mounted in an electronic device. For example, at least one ofthe data obtaining unit 141-1, the pre-processing unit 141-2, thelearning data selection unit 141-3, the model learning unit 141-4, andthe model evaluation unit 141-5 may be manufactured in the form of adedicated hardware chip for AI, or may be manufactured as a part of anexisting generic-purpose processor (for example, a CPU or an AP) or agraphic dedicated processor (for example, a GPU, an ISP), and may bemounted in the above-described various display apparatuses 100.

In addition, the data obtaining unit 141-1, the pre-processing unit141-2, the learning data selection unit 141-3, the model learning unit141-4, and the model evaluation unit 141-5 may be mounted in oneelectronic device, or may be respectively mounted in separate electronicdevices. For example, a portion of the data obtaining unit 141-1, thepre-processing unit 141-2, the learning data selection unit 141-3, themodel learning unit 141-4, and the model evaluation unit 141-5 may beincluded in the display apparatus 100, and the other portion may beincluded in the server 200.

At least one of the data obtaining unit 141-1, the pre-processing unit141-2, the learning data selection unit 141-3, the model learning unit141-4, and the model evaluation unit 141-5 may be implemented by using asoftware module. When at least one of the data obtaining unit 141-1, thepre-processing unit 141-2, the learning data selection unit 141-3, themodel learning unit 141-4, and the model evaluation unit 141-5 isimplemented by using a software module (or a program module includinginstructions), the software module may be stored in a non-transitorycomputer readable recording medium. At least one software module may beprovided by an OS or may be provided by a predetermined application.Alternatively, a portion of at least one software module may be providedby the OS, and the other portion may be provided by the predeterminedapplication.

FIG. 4B is a block diagram of a data recognition unit according to anembodiment of the present disclosure.

Referring to FIG. 4B, the data recognition unit 142 according to anembodiment may include a data obtaining unit 142-1 and a recognitionresult providing unit 142-4. In addition, the data recognition unit 142may further selectively include at least one of a pre-processing unit142-2, a recognition data selection unit 142-3, and a model update unit142-5.

The data obtaining unit 142-1 may obtain recognition data which isnecessary for determining a situation.

The recognition result providing unit 142-4 may determine a situation byapplying selected recognition data to the data recognition model. Therecognition result providing unit 142-4 may provide a recognition resultaccording to a purpose of recognition of inputted data. The recognitionresult providing unit 142-4 may apply selected data to the datarecognition model by using recognition data selected by the recognitiondata selection unit 142-3 as an input value. In addition, therecognition result may be determined by the data recognition model.

For example, the recognition result providing unit 142-4 may classifyselected image frames according to a classification criterion which isdetermined at the data recognition model. In addition, the recognitionresult providing unit 142-4 may output a classified characteristic valuesuch that the processor 140 generates a fingerprint. In another example,the recognition result providing unit 142-4 may apply the selected imageframes to the data recognition model, and may determine a genre of acontent to which the image frames belong. In response to the genre ofthe content being determined, the processor 140 may request fingerprintdata of granularity corresponding to the content genre from the server200.

According to various embodiments of the present disclosure, therecognition result providing unit 142-4 may obtain a result ofdetermining a type of video/audio data by using recognition data relatedto information of predetermined video/audio data as an input value.

The information of the video/audio data may include, for example, aposition of a current frame in the whole video/audio and a reproducingtime, which are information on the current frame. In addition, theinformation on the video/audio may include at least one of a video/audioname, a video/audio ID, a video/audio provider, video/audio seriesinformation, a genre, information on whether the video/audio is areal-time broadcast, information on whether the video/audio is a paidcontent.

The type of the video data may include, for example, drama,advertisement, movie, news, or the like. The type of the audio data mayinclude, for example, music, news, advertisement, or the like. However,the type of the audio/video data is not limited thereto.

According to various embodiments of the present disclosure, therecognition result providing unit 142-4 may obtain a result ofestimating a probability that the video/audio data is changed to anothervideo/audio data during production, or a result of estimatingvideo/audio data to be reproduced next time after the reproduction iscompleted, by using recognition data related to the information on thepredetermined video/audio data and the type of the video/audio data asan input value.

The data recognition unit 142 may further include the pre-processingunit 142-2, the recognition data selection unit 142-3, and the modelupdate unit 142-5 in order to improve the result of recognizing of thedata recognition model or in order to save resources or time necessaryfor providing the recognition result.

The pre-processing unit 142-2 may pre-process obtained data such thatthe obtained recognition data can be used to determine a situation. Thepre-processing unit 142-2 may process the obtained recognition data in apredetermined format such that the recognition result providing unit142-4 can use the obtained recognition data to determine a situation.

The recognition data selection unit 142-3 may select recognition datawhich is necessary for determining a situation from among thepre-processed data. The selected recognition data may be provided to therecognition result providing unit 142-4. The recognition data selectionunit 142-3 may select recognition data necessary for determining asituation from among the pre-processed data according to a predeterminedselection criterion for determining a situation. In addition, therecognition data selection unit 142-3 may select data according to apredetermined selection criterion according to learning by theabove-described model learning unit 141-4.

The model update unit 142-5 may control to update the data recognitionmodel based on evaluation on the recognition result provided by therecognition result providing unit 142-4. For example, the model updateunit 142-5 may provide the recognition result provided by therecognition result providing unit 142-4 to the model learning unit141-4, such that the model learning unit 141-4 controls to update thedata recognition model.

At least one of the data obtaining unit 142-1, the pre-processing unit142-2, the recognition data selection unit 142-3, the recognition resultproviding unit 142-4, and the model update unit 142-5 included in thedata recognition unit 142 may be manufactured in the form of at leastone hardware chip, and may be mounted in an electronic device. Forexample, at least one of the data obtaining unit 142-1, thepre-processing unit 142-2, the recognition data selection unit 142-3,the recognition result providing unit 142-4, and the model update unit142-5 may be manufactured in the form of a dedicated hardware chip forAI, or may be manufactured as a part of an existing generic-purposeprocessor (for example, a CPU or an AP) or a graphic dedicated processor(for example, a GPU, an ISP), and may be mounted in the above-describedvarious display apparatuses 100.

In addition, the data obtaining unit 142-1, the pre-processing unit142-2, the recognition data selection unit 142-3, the recognition resultproviding unit 142-4, and the model update unit 142-5 may be mounted inone electronic device, or may be respectively mounted in separateelectronic devices. For example, a portion of the data obtaining unit142-1, the pre-processing unit 142-2, the recognition data selectionunit 142-3, the recognition result providing unit 142-4, and the modelupdate unit 142-5 may be included in the display apparatus 100, and theother portion may be included in the server 200.

At least one of the data obtaining unit 142-1, the pre-processing unit142-2, the recognition data selection unit 142-3, the recognition resultproviding unit 142-4, and the model update unit 142-5 may be implementedby using a software module. When at least one of the data obtaining unit142-1, the pre-processing unit 142-2, the recognition data selectionunit 142-3, the recognition result providing unit 142-4, and the modelupdate unit 142-5 is implemented by using a software module (or aprogram module including instructions), the software module may bestored in a non-transitory computer readable recording medium. At leastone software module may be provided by an OS or may be provided by apredetermined application. Alternatively, a portion of at least onesoftware module may be provided by the OS, and the other portion may beprovided by the predetermined application.

FIG. 5 is a block diagram illustrating a configuration of a displayapparatus according to an embodiment of the present disclosure.

Referring to FIG. 5, the display apparatus 100 may include a display110, a memory 120, a communication unit 130, a processor 140, an imagereceiver 150, an image processor 160, an audio processor 170, and anaudio outputter 180.

The display 110 may display various image contents, information, a UI,or the like which are provided by the display apparatus 100.Specifically, the display 110 may display an image content and a UIwindow which are provided by an external device (for example, a set-topbox). For example, the UI window may include EPG, a menu for selecting acontent to be reproduced, content-related information, an additionalapplication execution button, a guide message, a notification message, afunction setting menu, a calibration setting menu, an operationexecution button, or the like. The display 110 may be implemented invarious forms, such as a liquid crystal display (LCD), an organic lightemitting diode (OLED), an active-matrix OLED (AM-OLED), a plasma displaypanel (PDP), or the like.

The memory 120 may store various programs and data necessary foroperations of the display apparatus 100. The memory 120 may beimplemented in the form of a flash memory, a hard disk, or the like. Forexample, the memory 120 may include a read only memory (ROM) for storinga program for performing operations of the display apparatus 100, and arandom access memory (RAM) for temporarily storing data which isaccompanied by the operations of the display apparatus 100. In addition,the memory 120 may further include an electrically erasable andprogrammable ROM (EEPROM) for storing various reference data.

The memory 120 may store a program and data for configuring variousscreens to be displayed on the display 110. In addition, the memory 120may store a program and data for performing a specific service. Forexample, the memory 120 may store a plurality of fingerprints, a viewinghistory, content information, or the like. The fingerprint may begenerated by the processor 140 or may be received from the server 200.

The communication unit 130 may communicate with the server 200 accordingto various types of communication methods. The communication unit 130may exchange fingerprint data with the server 200 connected thereto in awire or wireless manner. In addition, the communication unit 130 mayreceive, from the server 200, content information, a control signal forchanging a content recognition period, additional information,information on a product appearing in a content, or the like. Inaddition, the communication unit 130 may stream image data from anexternal server. The communication unit 130 may include variouscommunication chips for supporting wire/wireless communication. Forexample, the communication unit 130 may include a chip which operates ina wire local area network (LAN), wireless LAN (WLAN), Wi-Fi, Bluetooth(BT), or near field communication (NFC) method.

The image receiver 150 may receive image content data through varioussources. For example, the image receiver 150 may receive broadcastingdata from an external broadcasting station. In another example, theimage receiver 150 may receive image data from an external device (forexample, a set-top box, a digital versatile disc (DVD) player), or mayreceive image data which is streamed from an external server through thecommunication unit 130.

The image processor 160 performs image processing with respect to imagedata received from the image receiver 150. The image processor 160 mayperform various image processing operations, such as decoding, scaling,noise filtering, frame rate conversion, or resolution conversion, withrespect to the image data.

The audio processor 170 may perform processing with respect to audiodata. For example, the audio processor 170 may perform decoding,amplification, noise filtering, or the like with respect to audio data.

The audio outputter 180 may output not only various audio data processedat the audio processor, but also various notification sounds or voicemessages.

The processor 140 may control the above-described elements of thedisplay apparatus 100. For example, the processor 140 may receive afingerprint or content information through the communication unit 130.In addition, the processor 140 may adjust a content recognition periodusing the received content information. The processor 140 may beimplemented by using a single CPU to perform a control operation, asearch operation, or the like, and may be implemented by using aplurality of processors and an IP performing a specific function.

Hereinafter, the operation of the processor 140 will be described belowwith reference to drawings.

FIG. 6 is a view illustrating hybrid ACR according to an embodiment ofthe present disclosure.

Referring to FIG. 6, the hybrid ACR refers to a method of using acombination of local ACR, according to which the processor 140recognizes a reproduced content using fingerprint information stored inthe memory 120, and server ACR according to which a content isrecognized through information received from the server 200. When thelocal ACR and the server ACR are combined, a load on the server 200 canbe reduced and the display apparatus 100 can recognize what content isbeing reproduced with precision.

The processor 140 may recognize what content is being currentlyreproduced, and may adjust to perform ACR with appropriate precisionbased on the result of recognition. For example, the processor 140 mayadjust a content recognition period based on content information, andmay determine a content to be received from the server 200 in advanceand stored, and a quantity of fingerprints on the content.

Referring to FIG. 6, the processor 140 may extract a characteristic of ascreen of a currently reproduced content, and may generate afingerprint. In addition, the processor 140 may search whether there isa fingerprint matching the generated fingerprint from among a pluralityof fingerprints stored in the memory 120 (CD local ACR).

In response to a fingerprint matching the generated fingerprint beingsearched in the memory 120, the processor 140 may recognize thecurrently reproduced content based on information on the contentcorresponding to the searched fingerprint. For example, the informationon the content corresponding to the fingerprint may include a positionof a current frame in the whole content, a reproducing time, or thelike, which are information on the current frame. In addition, theinformation on the content corresponding to the fingerprint may includeat least one of a content name, a content ID, a content provider,content series information, a genre, information on whether the contentis a real-time broadcast, and information on whether the content is apaid content.

In response to the local ACR succeeding as described above, theprocessor 140 does not have to try to perform server ACR, and thus aload on the server 200 can be reduced. For appropriate local ACR, thememory 120 should store necessary fingerprint information and contentinformation. This will be described again below.

On the other hand, in response to the fingerprint matching the generatedfingerprint not being searched in the memory 120, the processor 140 maycontrol the communication unit 130 to transmit a query for requestinginformation on the currently reproduced content to the server 200 (CDserver ACR). For example, the query may include the generatedfingerprint, a viewing history, information on the display apparatus100, or the like.

The server 200 may establish a fingerprint database regarding variousimage contents in advance in case a server ACR request is received fromthe display apparatus 100. The server 200 may analyze image contents andmay extract characteristics of all image frames. In addition, the server200 may generate fingerprints for distinguishing image frames from oneanother using the extracted characteristics. The server 200 mayestablish the database using the generated fingerprints.

The server 200 may extract at least one fingerprint from the requestedquery. In addition, the server 200 may search the extracted fingerprintin the established database, and may recognize what content is beingcurrently reproduced by the display apparatus 100. The server 200 maytransmit recognized content information to the display apparatus 100. Inaddition, the server 200 may add the recognized content information to aviewing history of each display apparatus and may manage the viewinghistory.

In addition, the processor 140 may control the communication unit 130 toreceive the information on the currently reproduced content and thefingerprint of the currently reproduced content from the server 200 inresponse to the query. Herein, the fingerprint of the currentlyreproduced content may be a fingerprint regarding frames which arepositioned after a currently displayed frame in time in the wholecontent. Since the processor 140 knows time indicated by the position ofthe currently displayed frame in the whole content based on thefingerprint included in the query, the processor 140 may receive, fromthe server 200, a fingerprint on frames which are expected to bereproduced after the currently displayed frame.

To appropriately combine the local ACR and the server ACR, the processor140 may determine a fingerprint to be received from the server 200 inadvance for the local ACR based on at least one of the result of contentrecognition and the viewing history, and may determine whether to changea content recognition period.

According to an embodiment of the present disclosure, the processor 140may determine a type of the content based on the information on thecurrently reproduced content. In addition, the processor 140 may changethe content recognition period according to the determined type of thecontent. The type of the content may be classified according to acriterion, such as details of the content, a genre, information onwhether the content is a real-time broadcast, an importance, or thelike.

According to another embodiment of the present disclosure, the processor140 may estimate the type of the content using a data recognition modelwhich is set to estimate a type of a content based on information on thecontent. The data recognition model set to estimate a type of a contentmay be, for example, a data recognition model that learns to have acriterion for estimating a type of a content (for example, video/audiodata) based on learning data related to information of a content (forexample, video/audio data) and a type of a content (for example,video/audio data).

FIGS. 7A and 7B are views illustrating fingerprint information havingdifferent granularities according to an embodiment of the presentdisclosure.

Referring to FIGS. 7A and 7B, for example, in response to it beingrecognized that a news content or an advertisement content is beingreproduced, the processor 140 may reduce the content recognition period.Since the news content or the advertisement content frequently changesdetails thereof, the processor 140 may need to exactly recognize thereproduced content. For example, the processor 140 may set the contentrecognition period such that the processor 140 tries to recognize thecontent in every frame. In this case, as shown in FIG. 7A, the processor140 may request fingerprint information on all frames to be reproducedafter the currently displayed frame from the server 200.

In another example, in response to it being recognized that a broadcastprogram content or a movie content is being reproduced, the processor140 may increase the content recognition period. Regarding the broadcastprogram content or the movie content, the processor 140 does not need tograsp details included in each frame and may only grasp whether the samecontent is continuously reproduced. Accordingly, the processor 140 mayrequest that only the frame to be displayed at time corresponding to thecontent recognition period from among the frames to be reproduced afterthe currently displayed frame includes fingerprint information. In theexample of FIG. 7B, since fingerprint information is required only onceevery fourth frame, the granularity of the fingerprint is lower thanthat of FIG. 7A.

According to an embodiment of the present disclosure, the processor 140may determine the content recognition period by analyzing an applicationwhich is executed while the user of the display apparatus 100 is viewinga content. For example, in the case of a drama content, the processor140 may normally set the content recognition period to be long. However,in response to it being determined that there is a history that the userhas executed a shopping application while viewing a drama and hasshopped for PPL products, the processor 140 may reduce the contentrecognition period regarding the drama content. The processor 140 maydetermine which frame of the drama content shows a PPL product, and maycontrol the display 110 to display a relevant advertisement with thedrama content. In addition, the processor 140 may control the display110 to display a UI for immediately executing a shopping application.

As described above, the processor 140 may learn a criterion fordetermining the content recognition period based on viewing historyinformation and information on an executed additional application. Bydoing so, the processor 140 may personalize the criterion fordetermining the content recognition period to each user. When learningthe criterion for determining the content recognition period, theprocessor 140 may use a learning scheme by AI, such as theabove-described unsupervised learning.

According to an embodiment of the present disclosure, the processor 140may differently determine a quantity of fingerprints to be requestedfrom the server in advance according to the recognized content. Inaddition, the processor 140 may determine a quantity of fingerprintsregarding subsequent frames on the currently reproduced contentaccording to the recognized type of the content and the determinedcontent recognition period.

The number of frames to be received from the server 200 in advance andstored may vary according to the recognized type of the content. Forexample, in the case of VOD or DVR, the server 200 may have informationon all image frames, but in the case of a live broadcast, the server 200may only receive image information of a few seconds (for example,information of hundreds of image frames in the case of 60 Hz) before thedisplay apparatus 100 does. Since there are fingerprints correspondingto respective frames, the quantity of fingerprints that the displayapparatus 100 receives from the server 200 in advance and stores mayalso vary.

For example, in response to it being determined that the recognized typeof the content is a drama content and thus the content is set to berecognized every 30 seconds, the processor 140 may determine thequantity of fingerprints to be requested to 0. Since the server 200 doesnot have a fingerprint corresponding to a frame to be reproduced after30 seconds in a live broadcast, the processor 140 may omit anunnecessary communicating process.

According to an embodiment of the present disclosure, the processor 140may change the content recognition period based on information on therecognized content and the viewing history. The viewing history mayinclude a content that the user has viewed, a viewing time, and anadditional application which has been executed during viewing time.

FIG. 8 is a view illustrating viewing history information according toan embodiment of the present disclosure.

Referring to FIG. 8, the processor 140 may determine whether thecurrently reproduced content will be continuously reproduced or anothercontent will be reproduced by comparing the recognized current contentand the viewing history. In addition, the processor 140 may change thecontent recognition period according to a probability that anothercontent is reproduced. In addition, the processor 140 may requestinformation on a content that is expected to be reproduced next timefrom the server 200, and may receive information necessary for local ACRfrom the server 200 in advance. By doing so, a probability that serverACR is performed can be reduced, and thus a load on the server 200 canbe reduced, and also, the display apparatus 100 can recognize thecontent precisely.

According to various embodiments of the present disclosure, theprocessor 140 may estimate a probability that another content isreproduced or estimate a content which is expected to be reproduced nexttime, using a data recognition model which is set to estimate aprobability that another content is reproduced during production or toestimate a content which is expected to be reproduced next time afterthe reproduction is completed, based on the information of thereproduced content and the type of the content.

The data recognition model which is set to estimate a probability thatanother content is reproduced during reproduction or to estimate acontent which is expected to be reproduced next time after thereproduction is completed may estimate a probability that anothercontent is reproduced during reproduction or estimate a content which isexpected to be reproduced next time, based on learning data related tothe information of the content (for example, video/audio data), the typeof the content (for example, video/audio data), and a viewing history ofthe content (for example, video/audio data) (for example, a history ofhaving changed to another video/audio data).

For example, referring to the viewing history of FIG. 8, the user of thedisplay apparatus 100 usually views a news content on channel 3 from17:00 to 18:00. When a content currently recognized at 17:30 is a musiccontent on channel 2, the processor 140 may determine that there is ahigh probability that the channel is changed. Therefore, the processor140 may adjust the content recognition period to be short and mayfrequently check whether the reproduced content is changed.

According to an embodiment of the present disclosure, the processor 140may receive additional information related to the content from theserver 200 with the fingerprint. For example, the additional informationmay include a content name, a content reproducing time, a contentprovider, information on a PPL product appearing in a content, anadvertisement related to PPL production information, an executableadditional application, or the like.

FIG. 9 is a view illustrating display of additional information with acontent according to an embodiment of the present disclosure.

Referring to FIG. 9, for example, the processor 140 may receiveadditional information indicating that a PPL product 910 is included ina specific image frame. In addition, the processor 140 may control thedisplay 110 to display a UI 920 including the received additionalinformation with the content. The UI 920 may include a photo of the PPLproduct 910, a guide message, an additional application executionbutton, or the like.

According to the above-described embodiments of the present disclosure,the display apparatus 100 may reduce a load on the server 200 bydynamically adjusting the content recognition period, while performingACR precisely.

FIG. 10 is a flowchart illustrating a content recognizing method of adisplay apparatus according to an embodiment of the present disclosure.

Referring to FIG. 10, the display apparatus 100 may capture a screen ofa currently reproduced content, first. In addition, the displayapparatus 100 may extract a characteristic from the captured screen, andmay generate a fingerprint using the extracted characteristic atoperation S1010. The fingerprint is identification information fordistinguishing one image from the other images. Specifically, thefingerprint is characteristic data which is extracted from a video oraudio signal included in a frame.

According to various embodiments of the present disclosure, the displayapparatus 100 may generate the fingerprint using the data recognitionmodel described above with reference to FIGS. 3 to 4B.

The display apparatus 100 may search whether a fingerprint matching thegenerated fingerprint is stored in the display apparatus 100 atoperation S1020. For example, the display apparatus 100 may performlocal ACR first. In response to the local ACR succeeding, the displayapparatus 100 may recognize what content is currently reproduced withouttransmitting a query to the server 200. Since an inner storage space ofthe display apparatus 100 is limited, the display apparatus 100 shouldappropriately select fingerprint information to be received in advanceand stored.

The display apparatus 100 may determine whether to transmit a queryincluding the generated fingerprint to the external server 200 accordingto a result of searching, that is, a result of local ACR at operationS1030.

FIG. 11 is a flowchart illustrating a content recognizing method of adisplay apparatus according to an embodiment of the present disclosure.

Referring to FIG. 11, since operations S1110 and S1120 correspond tooperations S1010 and S1020, a redundant explanation is omitted.

In response to the fingerprint matching the generated fingerprint beingsearched in the display apparatus 100 at operation S1130-Y, the displayapparatus 100 may recognize the currently reproduced content using thestored fingerprint at operation S1140.

On the other hand, in response to the fingerprint matching the generatedfingerprint not being searched at operation S1130-N, the displayapparatus 100 may transmit a query for requesting information on thecurrently reproduced content to the external server 200 at operationS1150. The display apparatus 100 may receive content information fromthe server 200 at operation S1160. In addition, the display apparatus100 may further receive a fingerprint regarding a content which isexpected to be reproduced next time from the server 200. For example,the display apparatus 100 may receive a fingerprint regarding a framewhich is positioned after a currently reproduced frame in time in thewhole content, and a fingerprint regarding a frame of another contentwhich is expected to be reproduced next time.

As described above, the display apparatus 100 may recognize what contentis currently reproduced through local ACR or server ACR. Hereinafter, anoperation of the display apparatus after a content is recognized will bedescribed.

FIG. 12A is a view illustrating a method for changing a contentrecognition period of a display apparatus according to an embodiment ofthe present disclosure.

Referring to FIG. 12A, the display apparatus 100 may recognize acurrently reproduced content at operation S1210. In addition, thedisplay apparatus 100 may determine a type of the content usinginformation of the recognized content at operation S1220. For example,the type of the content may be classified based on a criterion, such asdetails of the content, a genre, information whether the content is areal-time broadcast, an importance, or the like. The criterion forclassifying the type of the content may be learned by the displayapparatus 100 itself by using AI (for example, the data recognitionmodel described in FIGS. 3 to 4B).

In addition, the display apparatus 100 may change a content recognitionperiod according to the determined type of the content at operationS1230. For example, in the case of a news content or an advertisementcontent which frequently changes details of the reproduced content, thedisplay apparatus 100 may set the content recognition period to beshort. In addition, in the case of a VOD content which just requires adecision on whether a currently reproduced content is continuouslyreproduced, the display apparatus 100 may set the content recognitionperiod to be long.

This criterion may vary according to a personal viewing taste. Thedisplay apparatus 100 may set a personalized criterion using a viewinghistory. The display apparatus 100 may learn this criterion by itselfusing an unsupervised learning method.

FIG. 12B is a view illustrating a method for changing a contentrecognition period of a display apparatus according to an embodiment ofthe present disclosure.

Referring to FIG. 12B, the first processor 140-1 may control to executeat least one application installed in the display apparatus 100. Forexample, the first processor 140-1 may capture an image displayed on thedisplay and generate a fingerprint, and may perform ACR.

The second processor 140-2 may estimate a type of a content using a datarecognition model. The data recognition model may be, for example, a setof algorithms for estimating a type of a content using information ofthe content and/or a fingerprint generated from the content, using aresult of statistical machine learning.

Referring to FIG. 12B, the first processor 140-1 may recognize acurrently reproduced content at operation S1240. The first processor140-1 may recognize a content using local ACR or server ACR, forexample.

The first processor 140-1 may transmit a result of content recognitionto the second processor 140-2 at operation S1245. For example, the firstprocessor 140-1 may transmit the result of content recognition to thesecond processor 140-2 to request the second processor 140-2 to estimatea type of the reproduced content.

The second processor 140-2 may estimate the type of the reproducedcontent using the data recognition model at operation S1250. Forexample, the data recognition model may estimate the type of the content(for example, video/audio data) based on learning data related toinformation of the content (for example, video/audio data) and the typeof the content (for example, video/audio data).

The second processor 140-2 may derive a recognition period of thecontent according to the estimated type of the content at operationS1255. For example, in the case of a news content or an advertisementcontent which frequently changes details of the reproduced content, thedisplay apparatus 100 may set the content recognition period to beshort. In addition, in the case of a VOD content which just requires adecision on whether a currently reproduced content is continuouslyreproduced, the display apparatus 100 may set the content recognitionperiod to be long.

The second processor 140-2 may transmit the derived content recognitionperiod to the first processor 140-1 at operation S1260. The firstprocessor 140-1 may change the content recognition period based on thereceived content recognition period at operation S1270.

According to various embodiments of the present disclosure, the firstprocessor 140-1 may receive the estimated type of the content from thesecond processor 140-2, and may perform at operation S1255.

FIG. 13A is a view illustrating a method for determining a quantity offingerprints to be requested by a display apparatus according to anembodiment of the present disclosure.

Referring to FIG. 13A, the display apparatus 100 may recognize acurrently reproduced content at operation S1310. In addition, thedisplay apparatus 100 may determine a type of the content usinginformation of the recognized content at operation S1320. For example,the display apparatus 100 may determine (or estimate) the type of thecontent using a data recognition model (for example, the datarecognition model described in FIG. 3 to FIG. 4B).

Similarly to the method for changing the content recognition periodaccording to the determined type of the content, the display apparatus100 may determine a quantity of fingerprints to be received from theserver 200 in advance at operation S1330. Since the number of imageframes existing in the server 200 varies according to the type of thecontent, the number of fingerprints corresponding to the respectiveframes and existing in the server 200 may also vary according to thetype of the content.

The display apparatus 100 may determine the quantity of fingerprints tobe received by considering a genre of the content, a viewing history,information on whether the content is a live broadcast, or the like. Inresponse to the determined quantity of fingerprints being received, thedisplay apparatus 100 may perform optimized local ACR while minimizingthe quantity of fingerprints stored in the display apparatus 100.

FIG. 13B is a view illustrating a method for determining a quantity offingerprints to be requested by a display apparatus according to anotherembodiment of the present disclosure.

Referring to FIG. 13B, the first processor 140-1 may control to executeat least one application installed in the display apparatus 100. Forexample, the first processor 140-1 may capture an image displayed on thedisplay and generate a fingerprint, and may perform ACR.

The second processor 140-2 may estimate a type of a content using a datarecognition model. The data recognition model may be, for example, a setof algorithms for estimating a type of a content using information ofthe content and a fingerprint generated from the content, using a resultof statistical machine learning.

Referring to FIG. 13B, the first processor 140-1 may recognize acurrently reproduced content at operation S1340.

The first processor 140-1 may transmit a result of content recognitionto the second processor 140-2 at operation S1345.

The second processor 140-2 may estimate the type of the reproducedcontent using the data recognition model at operation S1350. Forexample, the data recognition model may estimate the type of the content(for example, video/audio data) based on learning data related toinformation of the content (for example, video/audio data) and the typeof the content (for example, video/audio data).

The second processor 140-2 may derive a quantity of fingerprints to berequested from a server (for example, the server 200 of FIG. 1)according to the estimated type of the content at operation S1355. Sincethe number of image frames existing in the server 200 varies accordingto the type of the content, the number of fingerprints corresponding tothe respective frames and existing in the server 200 may also varyaccording to the type of the content.

The second processor 140-2 may transmit the derived quantity offingerprints to be requested to the first processor 140-1 at operation51360. The first processor 140-1 may determine the quantity offingerprints to be requested based on the received quantity offingerprints at operation S1365.

According to various embodiments of the present disclosure, the firstprocessor 140-1 may receive the estimated type of the content from thesecond processor 140-2, and may perform at operation S1355.

FIGS. 14A, 14B, 15A, and 15B are views illustrating a method forpredicting a content of a display apparatus according to variousembodiments of the present disclosure.

Referring to FIG. 14A, the display apparatus 100 may recognize acurrently reproduced content at operation S1410.

In addition, the display apparatus 100 may calculate a probability thatthe currently reproduced content is changed based on information on therecognized content and a viewing history at operation S1420. Forexample, the viewing history may include a channel that a user hasviewed, a viewing time, an ID of the display apparatus, userinformation, an executed additional application, or the like.

According to various embodiments of the present disclosure, the displayapparatus 100 may estimate the probability that the currently reproducedcontent is changed using a data recognition model (for example, the datarecognition model described in FIGS. 3 to 4B).

In addition, the display apparatus 100 may change a content recognitionperiod according to the calculated probability at operation S1430. Forexample, in response to it being determined that the user usually enjoysviewing a content different from the currently recognized content withreference to the viewing history, the display apparatus 100 maydetermine that there is a high probability that the currently reproducedcontent is changed. In this case, the display apparatus 100 may changethe content recognition period to be short.

On the other hand, in response to it being determined that the contentcorresponding to the usual viewing history is reproduced, the displayapparatus 100 may determine that there is a low probability that thecurrently reproduced content is changed. In this case, the displayapparatus 100 may change the content recognition period to be long.

FIG. 14B is a view illustrating a method for predicting a content andchanging a content recognition period in a display apparatus including afirst processor and a second processor according to an embodiment of thepresent disclosure.

Referring to FIG. 14B, the first processor 140-1 may control to executeat least one application installed in the display apparatus 100. Forexample, the first processor 140-1 may capture an image displayed on thedisplay and generate a fingerprint, and may perform ACR.

The second processor 140-2 may estimate a probability that a content ischanged using a data recognition model. The data recognition model maybe, for example, a set of algorithms for estimating a probability thatvideo/audio data is changed to another video/audio data duringreproduction, based on learning data related to information ofvideo/audio data, a type of video/audio data, and a viewing history ofvideo/audio data (for example, a history of having changed to anothervideo/audio data).

Referring to FIG. 14B, the first processor 140-1 may recognize acurrently reproduced content at operation S1440.

The first processor 140-1 may transmit a result of content recognitionto the second processor 140-2 at operation S1445.

The second processor 140-2 may estimate a probability that thereproduced content is changed using the data recognition model atoperation S1450.

The second processor 140-2 may derive a content recognition periodaccording to the estimated probability at operation S1455.

The second processor 140-2 may transmit the derived content recognitionperiod to the first processor 140-1 at operation S1460.

The first processor 140-1 may change the content recognition periodbased on the received content recognition period at operation S1465.

According to various embodiments of the present disclosure, the firstprocessor 140-1 may receive the estimated probability that the contentis changed from the second processor 140-2, and may perform at operationS1455.

Referring to FIG. 15A, the display apparatus 100 may recognize acurrently reproduced content at operation S1510. In addition, thedisplay apparatus 100 may predict a content to be reproduced next timebased on a viewing history at operation S1520. For example, in responseto the user of the display apparatus 100 having a viewing history ofusually viewing specific two channels, the display apparatus 100 maypredict contents to be reproduced in the two channels as a content to bereproduced next time.

According to various embodiments of the present disclosure, the displayapparatus 100 may estimate a content to be reproduced after thecurrently reproduced content using a data recognition model (forexample, the data recognition model described in FIGS. 3 to 4B).

The display apparatus 100 may request fingerprint information of thepredicted content from the server 200 at operation S1530. In addition,the display apparatus 100 may receive information on the predictedcontent from the server 200 in advance, and may store the same atoperation S1540.

The information on the predicted content which is transmitted to thedisplay apparatus 100 from the server 200 may include at least one ofinformation of the content currently reproduced in the display apparatus100, fingerprints of the currently reproduced content and of the contentpredicted as being reproduced next time, and a control signal forchanging the content recognition period of the display apparatus 100.For example, the display apparatus 100 may receive fingerprintinformation of the contents to be reproduced in the above-described twochannels from the server 200 in advance, and may store the same. Bydoing so, the display apparatus 100 may receive optimized fingerprintsto be used for local ACR.

FIG. 15B is a view illustrating a method for predicting a content andreceiving information on a predicted content in advance in a displayapparatus including a first processor and a second processor 140-2according to an embodiment of the present disclosure.

Referring to FIG. 15B, the first processor 140-1 may control to executeat least one application installed in the display apparatus 100. Forexample, the first processor 140-1 may capture an image displayed on thedisplay and generate a fingerprint, and may perform ACR.

The second processor 140-2 may estimate a probability that a content ischanged using a data recognition model. The data recognition model maybe, for example, an algorithm for estimating video/audio data to bereproduced after the reproduction is completed, based on learning datarelated to information of video/audio data, a type of video/audio data,and a viewing history of video/audio data (for example, a history ofhaving changed to another video/audio data).

Referring to FIG. 15B, the first processor 140-1 may recognize acurrently reproduced content at operation S1550.

The first processor 140-1 may transmit a result of content recognitionto the second processor 140-2 at operation S1555.

The second processor 140-2 may estimate a content to be reproduced afterthe currently reproduced content using the data recognition model atoperation S1560.

The second processor 140-2 may transmit the content to be reproducednext time to the first processor 140-1 at operation S1565.

The first processor 140-1 may request information on the predictedcontent from a server (for example, the server 200 of FIG. 1) atoperation S1570.

The first processor 140-1 may receive the information on the predictedcontent from the server (for example, the server 200 of FIG. 1), andstore the same at operation S1575.

According to various embodiments of the present disclosure, the secondprocessor 140-2 may perform at operation S1570.

FIG. 16 is a view illustrating data being learned and recognized by adisplay apparatus and a server interlocked with each other according toan embodiment of the present disclosure.

Referring to FIG. 16, the server 200 may learn a criterion forrecognizing a content and/or a criterion for estimating a type of acontent and/or a criterion for estimating a probability that a contentis changed, and the display apparatus 100 may set a criterion fordistinguishing image frames based on results of learning by the server200, and may a type of a content and a probability that a content ischanged.

In this case, a data learning unit 240 of the server 200 may include adata obtaining unit 240-1, a pre-processing unit 240-2, a learning dataselection unit 240-3, a model learning unit 240-4, and a modelevaluation unit 240-5. The data learning unit 240 may perform thefunction of the data learning unit 141 shown in FIG. 4A. The datalearning unit 240 of the server 200 may learn in order for a datarecognition model to have a criterion for analyzing a characteristic ofvideo/audio data. The server 200 may analyze a characteristic of eachcaptured frame according to the learned criterion, and may generate afingerprint.

The data learning unit 240 may determine what learning data will be usedto determine a characteristic of a screen (or frame) of a capturedcontent. In addition, the data learning unit 240 may learn a criterionfor extracting the characteristic of the captured content using thedetermined learning data. The data learning unit 240 may obtain data tobe used for learning, and may learn the criterion for analyzing thecharacteristic by applying the obtained data to the data recognitionmodel which will be described below.

According to various embodiments of the present disclosure, the datalearning unit 240 may learn in order for the data recognition model tohave a criterion for estimating a type of video/audio data based onlearning data related to information of predetermined video/audio dataand a type of video/audio data.

According to various embodiments of the present disclosure, the datalearning unit 240 may learn in order for the data recognition model tohave a criterion for estimating a probability that video/audio data ischanged to another video/audio data during reproduction or estimatingvideo/audio data to be reproduced next time after the reproduction iscompleted, based on learning data related to information ofpredetermined video/audio data, a type of video/audio data, and aviewing history of video/audio data (for example, a history of havingchanged to another video/audio data).

In addition, the recognition result providing unit 142-4 of the displayapparatus 100 may determine a situation by applying data selected by therecognition data selection unit 142-3 to the data recognition modelgenerated by the server 200. In addition, the recognition resultproviding unit 142-4 may receive the data recognition model generated bythe server 200 from the server 200, and may analyze an image ordetermine a type of a content using the received data recognition model.In addition, the model update unit 142-5 of the display apparatus 100may provide the analyzed image and the determined type of the content tothe model learning unit 240-4 of the server 200, such that the datarecognition model can be updated.

For example, the display apparatus 100 may use the data recognitionmodel which is generated by using computing power of the server 200. Inaddition, a plurality of display apparatuses 100 transmit the learned orrecognized data information to the server 200, such that the datarecognition model of the server 200 can be updated. In addition, each ofthe plurality of display apparatuses 100 transmits the learned orrecognized data information to the server 200, such that the server 200can generate a data recognition model personalized to each of thedisplay apparatuses 100.

FIG. 17 is a flowchart illustrating a content recognizing method of adisplay system according to an embodiment of the present disclosure.

Referring to FIG. 17, the display system 1000 may include the displayapparatus 100 and the server 200. FIG. 17 illustrates a pull methodwhich requests a fingerprint at the display apparatus 100.

First, the display apparatus 100 may capture a screen of a currentlyreproduced content at operation S1605. In addition, the displayapparatus 100 may analyze the captured screen and extract acharacteristic. The display apparatus 100 may generate a fingerprint fordistinguishing the captured screen from other image frames using theextracted characteristic at operation S1610.

The display apparatus 100 may perform local ACR to match the generatedfingerprint with a stored fingerprint at operation S1615. The case inwhich the currently reproduced content is recognized by the local ACRwill not be described. In response to the currently reproduced contentnot being recognized by the local ACR, the display apparatus 100 maytransmit a query including the generated fingerprint to the server atoperation S1625.

The server 200 may already analyze various contents and may establish afingerprint database at operation S1620. The server 200 may extract thefingerprint from the received query. In addition, the server 200 maymatch the extracted fingerprint with a plurality of fingerprints storedin the fingerprint database at operation S1630. The server 200 mayrecognize what the content questioned by the display apparatus 100 isthrough the matching fingerprint. The server 200 may transmitinformation on the recognized content and fingerprints on next imageframes of the recognized content to the display apparatus 100 atoperation S1635.

For example, the display apparatus 100 may transmit a query forrequesting information on a fingerprint generated at the server 200. Theserver 200 may generate a response to the received query using a queryAPI, and may provide the response. The query API may be an API whichsearches the fingerprint included in the query in the fingerprintdatabase and provides stored relevant information. In response to thequery being received, the query API of the server 200 may search whetherthe fingerprint included in the query exists in the fingerprintdatabase. In response to the fingerprint being searched, the query APImay transmit a name of the content, a position of the framecorresponding to the searched fingerprint in the whole content, areproducing time, or the like to the display apparatus 100 in responseto the query. In addition, the query API may transmit fingerprintscorresponding to frames which are positioned after the frame showing thefingerprint in time in the whole content to the display apparatus 100.

In addition, when the content reproduced at the display apparatus 100can be streamed through the server 200 (for example, VOD or a broadcastsignal), the server 200 may transmit the fingerprints to the displayapparatus 100 with the next frames of the recognized content. In thiscase, the fingerprints may be paired with the corresponding frames ofthe recognized contents and transmitted. For example, the fingerprintmay be provided in the form of one file added to the content, andinformation for mapping fingerprints and corresponding frames may beincluded in the fingerprints.

The display apparatus 100 may determine a genre of the content, animportance, or the like using the received content information. Inaddition, the display apparatus 100 may change a content recognitionperiod based on the criterion, such as the genre of the content, theimportance, or the like at operation S1640. In addition, the displayapparatus 100 may predict a content to be reproduced next time using aviewing history with the content information at operation S1645. Thecontent to be reproduced next time refers to a content which isdifferent from the currently reproduced content.

The display apparatus 100 may request a fingerprint on the predictedcontent from the server 200 at operation S1650. In addition, the displayapparatus 100 may request the predicted content itself as well as thefingerprint from the server 200. In response to this, the server 200 maytransmit the requested fingerprint to the display apparatus 100 atoperation S1655. In response to the predicted content being stored inthe server 200 or the predicted content being able to be streamed to thedisplay apparatus 100 through the server 200, the server 200 maytransmit not only the requested fingerprint but also the content pairedwith the fingerprint to the display apparatus 100.

The display apparatus 100 may store the received fingerprint and may usethe same for local ACR when a next content recognition period arrives.

FIG. 18 is a flowchart illustrating a content recognizing method of adisplay system according to an embodiment of the present disclosure.

Referring to FIG. 18, the display system 1000 may include the displayapparatus 100 and the server 200. FIG. 18 illustrates a push method inwhich the server 200 transmits a fingerprint preemptively.

First, the display apparatus 100 may capture a screen of a currentlyreproduced content at operation S1805. In addition, the displayapparatus 100 may analyze the captured screen and extract acharacteristic. The display apparatus 100 may generate a fingerprint fordistinguishing the captured screen from other image frames using theextracted characteristic at operation S1810.

The display apparatus 100 may perform local ACR to match the generatedfingerprint with a stored fingerprint at operation S1815. In addition,in response to the currently reproduced content not being recognized bythe local ACR, the display apparatus 100 may transmit a query includingthe generated fingerprint to the server at operation S1825.

The query may include information of the display apparatus 100 inaddition to the fingerprint. For example, the information of the displayapparatus 100 may be a physical ID of the display apparatus 100, an IPaddress of the display apparatus 100, or information for specifying theuser of the display apparatus 100, such as a user ID which istransmitted to the server 200 through the display apparatus 100.

The server 200 may manage a viewing history on each display apparatus100 using the information of the display apparatus 100. For example, inresponse to a client device accessing, the server 200 may collect adevice ID, and may perform the above-described operation using a clientmanagement API for managing a viewing history for each device ID.

The server 200 may already analyze various contents and may establish afingerprint database at operation S1820. The server 200 may storeinformation of contents corresponding to fingerprints in the database.For example, the server 200 may store, in the database, a name of acontent corresponding to a fingerprint, a position of a framecorresponding to a fingerprint in the whole content, a reproducing time,a content ID, a content provider, content series information, a genre,information on whether the content is a real-time broadcast, informationon whether the content is paid content, or the like.

The server 200 may extract the fingerprint from the received query. Forexample, the query API of the server 200 may extract only informationcorresponding to the fingerprint from string information of the receivedquery. In addition, the server 200 may match the extracted fingerprintwith a plurality of fingerprints stored in the fingerprint database atoperation 51830.

The server 200 may recognize what the content questioned by the displayapparatus 100 is through the matching fingerprint. The server 200 maydetermine a genre of the content, an importance, or the like using thedetermined content information. In addition, the server 200 may change acontent recognition period based on the criterion, such as the genre ofthe content, the importance, or the like at operation S1835. Inaddition, the server 200 may predict a content to be reproduced nexttime using a viewing history with the content information at operationS1840. For example, in the embodiment of FIG. 18, the server 200 mayperform the operations of changing the content recognition period andpredicting the content to be reproduced next time.

Based on the information grasped in this process, the server 200 maytransmit the fingerprint information or the like to the displayapparatus 100 without receiving a request from the display apparatus 100at operation S1845. For example, the server 200 may transmit, to thedisplay apparatus 100, information of the content currently reproducedat the display apparatus 100, fingerprints of the currently reproducedcontent and of the content predicted as being reproduced next time, anda control signal for changing the content recognition period of thedisplay apparatus 100. In another example, the server 200 may transmitthe content itself which is predicted as being reproduced next time withthe fingerprint. In this case, the fingerprints may be combined with allof the frames of the content, or the fingerprints may be combined withevery frame of intervals which are set according to the contentrecognition period.

In addition, the server 200 may transmit an advertisement screenregarding a product included in the frame of the content to be displayedon the display apparatus 100, a product buy UI, or the like to thedisplay apparatus 100 without receiving a request from the displayapparatus 100. The server 200 may grasp a history of having bought aproduct during viewing time based on the information of the displayapparatus 100 received from the display apparatus 100. The server 200may change a frequency of transmitting the advertisement screen or thelike using personalized information, such as a product buying history ora viewing history.

The server 200 may generate the advertisement screen with a sizesuitable to each screen displayed on the display apparatus 100, usingresolution information of the display apparatus 100, informationindicating which portion of the frame of the content corresponds to abackground, or the like. In addition, the server 200 may transmit acontrol signal for displaying the advertisement screen on an appropriateposition on the screen of the display apparatus 100 to the displayapparatus 100 with the advertisement screen.

In another example, the server 200 may request a second server 300 toprovide an advertisement screen. For example, the second server 300 maybe a separate server which provides an advertisement providing function.The second server 300 may receive information including a product to beadvertised, a resolution of the display apparatus 100, or the like fromthe server 200. According to the received information, the second server300 may generate an advertisement screen with an appropriate size. Thesecond server 300 may transmit the generated advertisement screen to theserver 200, or may directly transmit the advertisement screen to thedisplay apparatus 100. In the embodiment in which the second server 300directly transmits the advertisement screen to the display apparatus100, the server 200 may provide communication information, such as an IPaddress of the display apparatus 100 to the second server 300.

FIG. 19 is a view illustrating a situation in which a display apparatuschanges a content recognition period according to a probability that acontent is changed by interlocking with a server according to anembodiment of the present disclosure.

Referring to FIG. 19, the server 200 may estimate a probability that acontent is changed using a data recognition model. The data recognitionmodel may be, for example, a set of algorithms for estimating aprobability that video/audio data is changed to another video/audio dataduring reproduction, based on learning data related to information ofvideo/audio data, a type of video/audio data, and a viewing history ofvideo/audio data (for example, a history of having changed to anothervideo/audio data).

In this case, an interface for transmitting/receiving data between thedisplay apparatus 100 and the server 200 may be defined.

For example, an API having learning data to be applied to the datarecognition model as a factor value (or a parameter value or a transfervalue) may be defined. The API may be defined as a set of sub-routinesor functions which is called at one protocol (for example, a protocoldefined by the display apparatus 100) to perform certain processing ofanother protocol (for example, a protocol defined by the server 200).For example, through the API, an environment in which one protocol canperform an operation of another protocol may be provided.

Referring to FIG. 19, the display apparatus 100 may recognize acurrently reproduced content at operation S1910. The display apparatus100 may recognize a content using local ACR or server ACR, for example.

The display apparatus 100 may transmit a result of content recognitionto the server 200 at operation S1920. For example, the display apparatus100 may transmit the result of content recognition to the server 200 torequest the server 200 to estimate a probability that the reproducedcontent is changed.

The server 200 may estimate a probability that the reproduced content ischanged using the data recognition model at operation S1930.

The server 200 may derive a content recognition period according to theprobability that the content is changed at operation S1940. For example,in response to it being determined that the user usually enjoys viewinga content different from the currently recognized content with referenceto a viewing history based on a query history (for example, channelchange) requested to the server 200 by the display apparatus 100, thedisplay apparatus 100 may determine that there is a high probabilitythat the currently reproduced content is changed. In this case, thedisplay apparatus 100 may change the content recognition period to beshort.

On the other hand, in response to it being determined that the contentcorresponding to the usual viewing history is reproduced, the displayapparatus 100 may determine that there is a low probability that thecurrently reproduced content is changed. In this case, the displayapparatus 100 may change the content recognition period to be long.

The server 200 may transmit the derived content recognition period tothe display apparatus 100 at operation S1950. The display apparatus 100may change the content recognition period based on the received contentrecognition period at operation S1960.

FIG. 20 is a view illustrating a method by which a display apparatuspredicts a content to be reproduced next time and receives informationon a predicted content in advance by interlocking with a serveraccording to an embodiment of the present disclosure.

Referring to FIG. 20, the server 200 may estimate a probability that acontent is changed using the data recognition model. The datarecognition model may be, for example, an algorithm for estimatingvideo/audio data to be reproduced next time after reproduction iscompleted, based on learning data related to information of video/audiodata, a type of video/audio data, and a viewing history of video/audiodata (for example, a history of having changed to another video/audiodata).

The display apparatus 100 may recognize a currently reproduced contentat operation 52010.

The display apparatus 100 may transmit a result of content recognitionto the server 200 at operation 52020.

The server 200 may estimate a content to be reproduced after thecurrently reproduced content using the data recognition model atoperation 52030.

The server 200 may search information on the estimated content atoperation 52040.

The server 200 may transmit the information on the content to bereproduced next time to the display apparatus 100 at operation 52050.

The display apparatus 100 may receive the information on the estimatedcontent from the server 200, and may store the same at operation 52060.

FIG. 21 is a view illustrating a method by which a display apparatuspredicts a content to be reproduced next time and receives informationon the predicted content in advance by interlocking with a plurality ofservers according to an embodiment of the present disclosure.

Referring to FIG. 21, the first server 200 may estimate a probabilitythat a content is changed using the data recognition model. The datarecognition model may be, for example, an algorithm for estimatingvideo/audio data to be reproduced next time after reproduction iscompleted, based on learning data related to information of video/audiodata, a type of video/audio data, and a viewing history of video/audiodata (for example, a history of having changed to another video/audiodata).

The third server 201 may include a cloud server which stores informationon contents, for example.

The display apparatus 100 may recognize a currently reproduced contentat operation S2110.

The display apparatus 100 may transmit a result of content recognitionto the first server 200 at operation S2120.

The first server 200 may estimate a content to be reproduced after thecurrently reproduced content using the data recognition model atoperation S2130.

The first server 200 may transmit the estimated content to the secondserver to request the second server to search information at operationS2140.

The third server 201 may search information on the content at operationS2150.

The third server 201 may transmit the information on the content to bereproduced next time to the first server 200 at operation S2160. Inaddition, the first server 200 may transmit the information on thecontent to be reproduced next time to the display apparatus 100 atoperation S2170. However, according to various embodiments of thepresent disclosure, the third server 201 may transmit the information onthe content to be reproduced next time to the display apparatus 100.

The display apparatus 100 may receive the information on the estimatedcontent from the first server 200 or the third server 201, and may storethe same at operation S2180.

Certain aspects of the present disclosure can also be embodied ascomputer readable code on a non-transitory computer readable recordingmedium. A non-transitory computer readable recording medium is any datastorage device that can store data which can be thereafter read by acomputer system. Examples of the non-transitory computer readablerecording medium include a Read-Only Memory (ROM), a Random-AccessMemory (RAM), Compact Disc-ROMs (CD-ROMs), magnetic tapes, floppy disks,and optical data storage devices. The non-transitory computer readablerecording medium can also be distributed over network coupled computersystems so that the computer readable code is stored and executed in adistributed fashion. In addition, functional programs, code, and codesegments for accomplishing the present disclosure can be easilyconstrued by programmers skilled in the art to which the presentdisclosure pertains.

At this point it should be noted that the various embodiments of thepresent disclosure as described above typically involve the processingof input data and the generation of output data to some extent. Thisinput data processing and output data generation may be implemented inhardware or software in combination with hardware. For example, specificelectronic components may be employed in a mobile device or similar orrelated circuitry for implementing the functions associated with thevarious embodiments of the present disclosure as described above.Alternatively, one or more processors operating in accordance withstored instructions may implement the functions associated with thevarious embodiments of the present disclosure as described above. Ifsuch is the case, it is within the scope of the present disclosure thatsuch instructions may be stored on one or more non-transitory processorreadable mediums. Examples of the processor readable mediums include aROM, a RAM, CD-ROMs, magnetic tapes, floppy disks, and optical datastorage devices. The processor readable mediums can also be distributedover network coupled computer systems so that the instructions arestored and executed in a distributed fashion. In addition, functionalcomputer programs, instructions, and instruction segments foraccomplishing the present disclosure can be easily construed byprogrammers skilled in the art to which the present disclosure pertains.

According to various embodiments of the present disclosure, thedisclosed embodiments may be implemented by using an S/W programincluding instructions stored in a computer-readable storage medium.

A computer is a device which calls stored instructions from a storagemedium, and can perform operations according to the disclosedembodiments according the called instructions, and may include thedisplay apparatus according to the disclosed embodiments.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. Herein, the “non-transitory” only meansthat the storage medium does not include signals and is tangible, anddoes not consider whether data is stored in the storage mediumsemi-permanently or temporarily.

In addition, the control method according to the disclosed embodimentsmay be included in a computer program product and provided. The computerprogram product may be traded between a seller and a purchaser as aproduct.

The computer program product may include an S/W program, and a computerreadable storage medium which stores the S/W program. For example, thecomputer program product may include a product in an S/W program form(for example, a downloadable application) which is electronicallydistributed through the manufacturer of the display apparatus or anelectronic market (for example, Google play store, App store). To beelectronically distributed, at least a part of the S/W program may bestored in a storage medium or may be temporarily generated. In thiscase, the storage medium may be a storage medium of a server of themanufacturer, a server of the electronic market, or an intermediateserver which temporarily stores the S/W program.

The computer program product may include a storage medium of a server ora storage medium of a device in a system which includes a server and adisplay apparatus. Alternatively, when there is a third device (forexample, a smart phone) communication connected with the server or thedisplay apparatus, the computer program product may include a storagemedium of the third device. Alternatively, the computer program productmay include an S/W program itself that is transmitted from the server tothe display apparatus or the third device, or transmitted from the thirddevice to the display apparatus.

In this case, one of the server, the display apparatus, and the thirddevice may execute the computer program product and perform the methodaccording to the disclosed embodiments. Alternatively, two or more ofthe server, the display apparatus, and the third device may execute thecomputer program product and perform the method according to thedisclosed embodiments in a distributed manner.

For example, the server (for example, a cloud server or an AI server)may execute the computer program product stored in the server, and maycontrol the display apparatus communication connected with the server toperform the method according to the disclosed embodiments.

In another example, the third device may execute the computer programproduct, and may control the display apparatus communication connectedwith the third device to perform the method according to the disclosedembodiments. When the third device executes the computer programproduct, the third device may download the computer program product fromthe server, and may execute the downloaded computer program product.Alternatively, the third device may execute the computer program productprovided in a preloaded state, and may perform the method according tothe disclosed embodiments.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present disclosure asdefined by the appended claims and their equivalents.

What is claimed is:
 1. A display apparatus comprising: a display; amemory storing a fingerprint which is obtained based on a characteristicof a content, and information regarding the content corresponding to thefingerprint; a transceiver; and at least one processor configured to:obtain a fingerprint based on a characteristic of a screen of a contentcurrently reproduced on the display, search presence/absence of a storedfingerprint matching the obtained fingerprint in the memory, identify,based on a result of the searching, whether to transmit a querycomprising the obtained fingerprint to a server through the transceiverto request information on the currently reproduced content, identify atype of the currently reproduced content using a data recognition modelthat uses the obtained fingerprint, and change a content recognitionperiod based on the identified type of the currently reproduced content.2. The display apparatus of claim 1, wherein the at least one processoris further configured to: recognize, based on the stored fingerprintmatching the obtained fingerprint being found in the memory, thecurrently reproduced content based on information on a contentcorresponding to the searched fingerprint, and control, based on thestored fingerprint matching the obtained fingerprint not being found inthe memory, the transceiver to transmit the query comprising thefingerprint to the server to request the information on the currentlyreproduced content.
 3. The display apparatus of claim 2, wherein the atleast one processor is further configured to control, based on thestored fingerprint matching the obtained fingerprint not being found inthe memory, the transceiver to receive the information on the currentlyreproduced content and one or more additional fingerprints of thecurrently reproduced content from the server.
 4. The display apparatusof claim 1, wherein the at least one processor is further configured to:recognize a content in every first period based on the currentlyreproduced content being an advertisement content, and recognize acontent in every second period which is longer than the first periodbased on the currently reproduced content being a broadcast programcontent.
 5. The display apparatus of claim 3, wherein the at least oneprocessor is further configured to: change a quantity of fingerprints ofthe currently reproduced content to be received according to theidentified type of the content.
 6. The display apparatus of claim 3,wherein the at least one processor is further configured to: calculate aprobability that the currently reproduced content is changed based onthe information on the currently reproduced content and a viewinghistory, and change a content recognition period according to thecalculated probability.
 7. The display apparatus of claim 2, wherein theat least one processor is further configured to: predict a content to bereproduced next time based on a viewing history, and request informationon the predicted content from the server.
 8. The display apparatus ofclaim 3, wherein the at least one processor is further configured to:receive additional information related to the currently reproducedcontent from the server, and control the display to display the receivedadditional information with the currently reproduced content.
 9. Amethod for recognizing a content of a display apparatus, the methodcomprising: obtaining a fingerprint based on a characteristic of ascreen of a currently reproduced content; searching whether a storedfingerprint matching the obtained fingerprint is stored in the displayapparatus; identifying, based on a result of the searching, whether totransmit a query comprising the obtained fingerprint to a server torequest information on the currently reproduced content; identifying atype of the currently reproduced content using a data recognition modelthat uses the obtained fingerprint; and changing a content recognitionperiod based on the identified type of the currently reproduced content.10. The method of claim 9, wherein the identifying whether to transmitthe query to the external server comprises: recognizing, based on thestored fingerprint matching the obtained fingerprint being found in thedisplay apparatus, the currently reproduced content based on informationon a content corresponding to the stored fingerprint, and transmitting,based on the stored fingerprint matching the obtained fingerprint notbeing found in the display apparatus, the query comprising thefingerprint to the server for requesting the information on thecurrently reproduced content.
 11. The method of claim 10, furthercomprising receiving, based on the stored fingerprint matching theobtained fingerprint not being found in the display apparatus, theinformation on the currently reproduced content and fingerprints of thecurrently reproduced content from the server.
 12. The method of claim 9,wherein the changing of the content recognition period comprises:recognizing a content in every first period based on the currentlyreproduced content being an advertisement content, and recognizing acontent in every second period which is longer than the first periodbased on the currently reproduced content being a broadcast programcontent.
 13. The method of claim 11, further comprising: changing aquantity of fingerprints of the currently reproduced content to bereceived according to the identified type of the content.
 14. The methodof claim 11, further comprising: calculating a probability that thecurrently reproduced content is changed based on the information on thecurrently reproduced content and a viewing history; and changing acontent recognition period according to the calculated probability. 15.The method of claim 10, further comprising: predicting a content to bereproduced next time based on a viewing history; and requestinginformation on the predicted content from the server.
 16. The method ofclaim 11, further comprising: receiving additional information relatedto the currently reproduced content from the server; and displaying thereceived additional information with the currently reproduced content.